easycv.file package

Submodules

easycv.file.base module

class easycv.file.base.IOBase

Bases: object

static register(options)

open(path: str, mode: str = 'r')

exists(path: str) → bool

move(src: str, dst: str)

copy(src: str, dst: str)

copytree(src: str, dst: str)

makedirs(path: str, exist_ok=True)

remove(path: str)

rmtree(path: str)

listdir(path: str, recursive=False, full_path=False, contains=None)

isdir(path: str) → bool

isfile(path: str) → bool

abspath(path: str) → str

last_modified(path: str) → datetime.datetime

last_modified_str(path: str) → str

size(path: str) → int

md5(path: str) → str

re_remote = re.compile('(oss|https?)://')

islocal(path: str) → bool

is_writable(path)

class easycv.file.base.IOLocal

Bases: easycv.file.base.IOBase

open(path, mode='r')

exists(path)

move(src, dst)

copy(src, dst)

copytree(src, dst)

makedirs(path, exist_ok=True)

remove(path)

rmtree(path)

listdir(path, recursive=False, full_path=False, contains: Optional[Union[List[str], str]] = None)

isdir(path)

isfile(path)

glob(path)

abspath(path)

last_modified(path)

last_modified_str(path)

size(path: str) → int

easycv.file.file_io module

easycv.file.file_io.set_oss_env(ak_id: str, ak_secret: str, hosts: Union[str, List[str]], buckets: Union[str, List[str]])

class easycv.file.file_io.IO(max_retry=10, retry_wait=0.1, max_retry_wait=30)

Bases: easycv.file.base.IOLocal

IO module to support both local and oss io. If access oss file, you need to authorize OSS, please refer to IO.access_oss.

__init__(max_retry=10, retry_wait=0.1, max_retry_wait=30)

Initialize self. See help(type(self)) for accurate signature.

access_oss(ak_id: str = '', ak_secret: str = '', hosts: Union[str, List[str]] = '', buckets: Union[str, List[str]] = '')

If access oss file, you need to authorize OSS as follows:

Method1: from easycv.file import io io.access_oss(

ak_id=’your_accesskey_id’, ak_secret=’your_accesskey_secret’, hosts=’your endpoint’ or [‘your endpoint1’, ‘your endpoint2’], buckets=’your bucket’ or [‘your bucket1’, ‘your bucket2’])

Method2:

Add oss config to your local file ~/.ossutilconfig, as follows: More oss config information, please refer to: https://help.aliyun.com/document_detail/120072.html ``` [Credentials]

language = CH endpoint = your endpoint accessKeyID = your_accesskey_id accessKeySecret = your_accesskey_secret

[Bucket-Endpoint]

bucket1 = endpoint1 bucket2 = endpoint2

``` Then run the following command, the config file will be read by default to authorize oss.

from easycv.file import io io.access_oss()

open(full_path, mode='r')

Same usage as the python build-in open. Support local path and oss path.

Example

from easycv.file import io io.access_oss(your oss config) # only oss file need, refer to IO.access_oss # Write something to a oss file. with io.open(‘oss://bucket_name/demo.txt’, ‘w’) as f:

f.write(“test”)

# Read from a oss file. with io.open(‘oss://bucket_name/demo.txt’, ‘r’) as f:

print(f.read())

Parameters

full_path – absolute oss path

exists(path)

Whether the file exists, same usage as os.path.exists. Support local path and oss path.

Example

from easycv.file import io io.access_oss(your oss config) # only oss file need, refer to IO.access_oss ret = io.exists(‘oss://bucket_name/dir’) print(ret)

Parameters

path – oss path or local path

move(src, dst)

Move src to dst, same usage as shutil.move. Support local path and oss path.

Example

from easycv.file import io io.access_oss(your oss config) # only oss file need, refer to IO.access_oss # move oss file to local io.move(‘oss://bucket_name/file.txt’, ‘/your/local/path/file.txt’) # move oss file to oss io.move(‘oss://bucket_name/file.txt’, ‘oss://bucket_name/file.txt’) # move local file to oss io.move(‘/your/local/file.txt’, ‘oss://bucket_name/file.txt’) # move directory io.move(‘oss://bucket_name/dir1’, ‘oss://bucket_name/dir2’)

Parameters

• src – oss path or local path

• dst – oss path or local path

safe_copy(src, dst, try_max=3)

oss always bug, we need safe_copy!

copy(src, dst)

Copy a file from src to dst. Same usage as shutil.copyfile. If you want to copy a directory, please use easycv.io.copytree

Example

from easycv.file import io io.access_oss(your oss config) # only oss file need, refer to IO.access_oss # Copy a file from local to oss: io.copy(‘/your/local/file.txt’, ‘oss://bucket/dir/file.txt’)

# Copy a oss file to local: io.copy(‘oss://bucket/dir/file.txt’, ‘/your/local/file.txt’)

# Copy a file from oss to oss:: io.copy(‘oss://bucket/dir/file.txt’, ‘oss://bucket/dir/file2.txt’)

Parameters

• src – oss path or local path

• dst – oss path or local path

copytree(src, dst)

Copy files recursively from src to dst. Same usage as shutil.copytree. If you want to copy a file, please use easycv.io.copy.

Example

from easycv.file import io io.access_oss(your oss config) # only oss file need, refer to IO.access_oss # copy files from local to oss io.copytree(src=’/your/local/dir1’, dst=’oss://bucket_name/dir2’) # copy files from oss to local io.copytree(src=’oss://bucket_name/dir2’, dst=’/your/local/dir1’) # copy files from oss to oss io.copytree(src=’oss://bucket_name/dir1’, dst=’oss://bucket_name/dir2’)

Parameters

• src – oss path or local path

• dst – oss path or local path

listdir(path, recursive=False, full_path=False, contains: Optional[Union[List[str], str]] = None)

List all objects in path. Same usage as os.listdir.

Example

from easycv.file import io io.access_oss(your oss config) # only oss file need, refer to IO.access_oss ret = io.listdir(‘oss://bucket/dir’, recursive=True) print(ret)

Parameters

• path – local file path or oss path.

• recursive – If False, only list the top level objects. If True, recursively list all objects.

• full_path – if full path, return files with path prefix.

• contains – substr to filter list files.

return: A list of path.

remove(path)

Remove a file or a directory recursively. Same usage as os.remove or shutil.rmtree.

Example

from easycv.file import io io.access_oss(your oss config) # only oss file need, refer to IO.access_oss # Remove a oss file io.remove(‘oss://bucket_name/file.txt’)

# Remove a oss directory io.remove(‘oss://bucket_name/dir/’)

Parameters

path – local or oss path, file or directory

rmtree(path)

Remove directory recursively, same usage as shutil.rmtree

Example

from easycv.file import io io.access_oss(your oss config) # only oss file need, refer to IO.access_oss io.remove(‘oss://bucket_name/dir_name’) # Or io.remove(‘oss://bucket_name/dir_name/’)

Parameters

path – oss path

makedirs(path, exist_ok=True)

Create directories recursively, same usage as os.makedirs

Example

from easycv.file import io io.access_oss(your oss config) # only oss file need, refer to IO.access_oss io.makedirs(‘oss://bucket/new_dir/’)

Parameters

path – local or oss dir path

isdir(path)

Return whether a path is directory, same usage as os.path.isdir

Example

from easycv.file import io io.access_oss(your oss config) # only oss file need, refer to IO.access_oss io.isdir(‘oss://bucket/dir/’)

Parameters

path – local or oss path

Return: bool, True or False.

isfile(path)

Return whether a path is file object, same usage as os.path.isfile

Example

from easycv.file import io io.access_oss(your oss config) # only oss file need, refer to IO.access_oss io.isfile(‘oss://bucket/file.txt’)

Parameters

path – local or oss path

Return: bool, True or False.

glob(file_path)

Return a list of paths matching a pathname pattern. .. rubric:: Example

from easycv.file import io io.access_oss(your oss config) # only oss file need, refer to IO.access_oss io.glob(‘oss://bucket/dir/*.txt’)

Parameters

path – local or oss file pattern

Return: list, a list of paths.

abspath(path)

authorize(path)

last_modified(path)

last_modified_str(path)

size(path: str) → int

Get the size of file path, same usage as os.path.getsize

Example

from easycv.file import io io.access_oss(your oss config) # only oss file need, refer to IO.access_oss size = io.size(‘oss://bucket/file.txt’) print(size)

Parameters

path – local or oss path.

Return: size of file in bytes

class easycv.file.file_io.OSSFile(bucket, path, position=0)

Bases: object

__init__(bucket, path, position=0)

Initialize self. See help(type(self)) for accurate signature.

write(content)

flush(retry=0)

close()

seek(position)

class easycv.file.file_io.BinaryOSSFile(bucket, path)

Bases: object

__init__(bucket, path)

Initialize self. See help(type(self)) for accurate signature.

class easycv.file.file_io.NullContextWrapper(obj)

Bases: object

__init__(obj)

Initialize self. See help(type(self)) for accurate signature.

easycv.file.utils module

easycv.file.utils.create_namedtuple(**kwargs)

easycv.file.utils.is_oss_path(s)

easycv.file.utils.is_url_path(s)

easycv.file.utils.url_path_exists(url)

easycv.file.utils.get_oss_config()

Get oss config file from env OSS_CONFIG_FILE, default file is ~/.ossutilconfig.

easycv.file.utils.oss_progress(desc)

easycv.file.utils.ignore_oss_error(msg='')

easycv.file.utils.mute_stderr()