easycv.datasets.utils package

Submodules

easycv.datasets.utils.tfrecord_util module

easycv.datasets.utils.tfrecord_util.get_imagenet_dali_tfrecord_feature()

easycv.datasets.utils.tfrecord_util.get_path_and_index(file_list_or_path)

easycv.datasets.utils.tfrecord_util.download_tfrecord(file_list_or_path, target_path, slice_count=1, slice_id=0, force=False)

Download data from oss. Use the processes on the gpus to slice download, each gpu process downloads part of the data. The number of slices is the same as the number of gpu processes. Support tfrecord of ImageNet style. tfrecord_dir

|—train1 |—train1.idx |—train2 |—train2.idx |—…

Parameters

• file_list_or_path – A list of absolute data path or a path str type(file_list) == list means this is the list type(file_list) == str means open(file_list).readlines()

• target_path – A str, download path

• slice_count – Download worker num

• slice_id – Download worker ID

• force – If false, skip download if the file already exists in the target path. If true, recopy and replace the original file.

Returns

list of str, download tfrecord path index_path: list of str, download tfrecord idx path

Return type

path

easycv.datasets.utils.type_util module

easycv.datasets.utils.type_util.is_dali_dataset_type(type_name)