Skip to content

cloneFolder

Short Description

The purpose of the cloneFolder function is to copy user actions from one folder to another. For example, if a user manually arranges thumbnails in the localNorm folder, this function can replicate those changes to the raw thumbnails.

Function

cloneFolder(copyFolder, applyFolder, TruePos='TruePos', TrueNeg='TrueNeg', PosToNeg='PosToNeg', NegToPos='NegToPos', verbose=True)

Parameters:

Name Type Description Default
copyFolder list

List of folders from which the user wants to replicate the file structure.

 required
applyFolder list

List of folders where the replicated file structure should be applied, in the same order as the copyFolder list.

 required
TruePos str

Name of the folder that holds the Thumbnails classified as True Positive.

 'TruePos'
TrueNeg str

Name of the folder that holds the Thumbnails classified as True Negative.

 'TrueNeg'
PosToNeg str

Name of the folder that holds the Thumbnails that were moved from True Positive to True Negative.

 'PosToNeg'
NegToPos str

Name of the folder that holds the Thumbnails that were moved from True Negative to True Positive.

 'NegToPos'
verbose bool

If True, print detailed information about the process to the console.

True

Returns:

Name Type Description
folder cloned folders

The file structure of the source Folder is replicated in the destination Folder.
Example 
# High level working directory
projectDir = '/Users/aj/Documents/cspotExampleData'

# list of folders to copy settings from
copyFolder = [projectDir + '/CSPOT/Thumbnails/localNorm/CD3D',
              projectDir + '/CSPOT/Thumbnails/localNorm/ECAD']
# list of folders to apply setting to
applyFolder = [projectDir + '/CSPOT/Thumbnails/CD3D',
               projectDir + '/CSPOT/Thumbnails/ECAD']
# note: Every copyFolder should have a corresponding applyFolder. The order matters! 

# The function accepts the four pre-defined folders. If you had renamed them, please change it using the parameter below.
cs.cloneFolder (copyFolder, 
                applyFolder, 
                TruePos='TruePos', TrueNeg='TrueNeg', 
                PosToNeg='PosToNeg', NegToPos='NegToPos')


# Same function if the user wants to run it via Command Line Interface
python cloneFolder.py             --copyFolder /Users/aj/Desktop/cspotExampleData/CSPOT/Thumbnails/localNorm/CD3D /Users/aj/Desktop/cspotExampleData/CSPOT/Thumbnails/localNorm/ECAD             --applyFolder /Users/aj/Desktop/cspotExampleData/CSPOT/Thumbnails/CD3D /Users/aj/Desktop/cspotExampleData/CSPOT/Thumbnails/ECAD
Source code in cspot/cloneFolder.py 
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
def cloneFolder (copyFolder, 
                 applyFolder,
                 TruePos='TruePos', 
                 TrueNeg='TrueNeg',
                 PosToNeg='PosToNeg', 
                 NegToPos='NegToPos',
                 verbose=True):
    """
Parameters:
    copyFolder (list):
        List of folders from which the user wants to replicate the file structure.

    applyFolder (list):
        List of folders where the replicated file structure should be applied,
        in the same order as the `copyFolder` list.

    TruePos (str, optional):
        Name of the folder that holds the Thumbnails classified as True Positive.

    TrueNeg (str, optional):
        Name of the folder that holds the Thumbnails classified as True Negative.

    PosToNeg (str, optional):
        Name of the folder that holds the Thumbnails that were moved from `True Positive`
        to `True Negative`.

    NegToPos (str, optional):
        Name of the folder that holds the Thumbnails that were moved from `True Negative`
        to `True Positive`.

    verbose (bool, optional):
        If True, print detailed information about the process to the console.  

Returns:
    folder (cloned folders):  
        The file structure of the source Folder is replicated in the destination Folder.

Example:
        ```python

        # High level working directory
        projectDir = '/Users/aj/Documents/cspotExampleData'

        # list of folders to copy settings from
        copyFolder = [projectDir + '/CSPOT/Thumbnails/localNorm/CD3D',
                      projectDir + '/CSPOT/Thumbnails/localNorm/ECAD']
        # list of folders to apply setting to
        applyFolder = [projectDir + '/CSPOT/Thumbnails/CD3D',
                       projectDir + '/CSPOT/Thumbnails/ECAD']
        # note: Every copyFolder should have a corresponding applyFolder. The order matters! 

        # The function accepts the four pre-defined folders. If you had renamed them, please change it using the parameter below.
        cs.cloneFolder (copyFolder, 
                        applyFolder, 
                        TruePos='TruePos', TrueNeg='TrueNeg', 
                        PosToNeg='PosToNeg', NegToPos='NegToPos')


        # Same function if the user wants to run it via Command Line Interface
        python cloneFolder.py \
            --copyFolder /Users/aj/Desktop/cspotExampleData/CSPOT/Thumbnails/localNorm/CD3D /Users/aj/Desktop/cspotExampleData/CSPOT/Thumbnails/localNorm/ECAD \
            --applyFolder /Users/aj/Desktop/cspotExampleData/CSPOT/Thumbnails/CD3D /Users/aj/Desktop/cspotExampleData/CSPOT/Thumbnails/ECAD

        ```

    """

    #TruePos='TruePos'; TrueNeg='TrueNeg'; PosToNeg='PosToNeg'; NegToPos='NegToPos'

    # Convert the path to list
    if isinstance (copyFolder, str):
        copyFolder = [copyFolder]
    if isinstance (applyFolder, str):
        applyFolder = [applyFolder]

    # Quick Check!
    if len(copyFolder) is not len(applyFolder):
        raise ValueError('The number of copyFolder and applyFolder should match, please check!' )

    # function to delete images
    def deleteFile(files, location):
        for f in files:
            # full path
            #full_path = location  + f
            full_path = pathlib.Path.joinpath(location, f)
            if os.path.exists(full_path):
                os.remove(full_path)

    # Function to move images
    def moveFile(files, from_loc, to_loc):
        for f in files:
            # full path
            full_path_from = pathlib.Path.joinpath(from_loc, f) # from_loc + f
            full_path_to = pathlib.Path.joinpath(to_loc, f) # to_loc + f
            # move file
            if os.path.exists(full_path_from):
                shutil.move(full_path_from, full_path_to)

    # path lib of all folder
    all_folders = [pathlib.Path(p) for p in copyFolder]

    # copy from location
    pos_aug_location = [pathlib.Path(p + '/' + str(TruePos)) for p in copyFolder]
    neg_aug_location = [pathlib.Path(p + '/' + str(TrueNeg)) for p in copyFolder]
    pos2neg_aug_location = [pathlib.Path(p + '/' + str(PosToNeg)) for p in copyFolder]
    neg2pos_aug_location = [pathlib.Path(p + '/' + str(NegToPos)) for p in copyFolder]

    # copy to location
    pos_real_location = [pathlib.Path(p + '/' + str(TruePos)) for p in applyFolder]
    neg_real_location = [pathlib.Path(p + '/' + str(TrueNeg)) for p in applyFolder]
    pos2neg_real_location = [pathlib.Path(p + '/' + str(PosToNeg)) for p in applyFolder]
    neg2pos_real_location = [pathlib.Path(p + '/' + str(NegToPos)) for p in applyFolder]



    # function
    def processFolder (folderIndex):
        if verbose is True:
            print ('Processing: ' + str(all_folders[folderIndex].stem))

        # create a list of all file names in the applyFolder
        pos_files = next(walk(pos_real_location[folderIndex]), (None, None, []))[2]
        neg_files = next(walk(neg_real_location[folderIndex]), (None, None, []))[2]

        # Find file names within each of the copyFolder
        pos = next(walk(pos_aug_location[folderIndex]), (None, None, []))[2]
        neg = next(walk(neg_aug_location[folderIndex]), (None, None, []))[2]
        pos2neg = next(walk(pos2neg_aug_location[folderIndex]), (None, None, []))[2]
        neg2pos = next(walk(neg2pos_aug_location[folderIndex]), (None, None, []))[2]  # [] if no file

        # Find images to delete
        pos_del = list(set(pos_files).difference(pos + pos2neg))
        neg_del = list(set(neg_files).difference(neg + neg2pos))

        # delete files
        deleteFile(files=pos_del, location=pos_real_location[folderIndex])
        deleteFile(files=neg_del, location=neg_real_location[folderIndex])

        # move files
        moveFile (files=pos2neg, from_loc=pos_real_location[folderIndex], to_loc=pos2neg_real_location[folderIndex])
        moveFile (files=neg2pos, from_loc=neg_real_location[folderIndex], to_loc=neg2pos_real_location[folderIndex])

        # print the number of files
        posaug = len(next(walk(pos_aug_location[folderIndex]), (None, None, []))[2])
        posreal = len(next(walk(pos_real_location[folderIndex]), (None, None, []))[2])
        negaug = len(next(walk(neg_aug_location[folderIndex]), (None, None, []))[2])
        negreal = len(next(walk(neg_real_location[folderIndex]), (None, None, []))[2])
        postonegaug = len(next(walk(pos2neg_aug_location[folderIndex]), (None, None, []))[2])
        postonegreal = len(next(walk(pos2neg_real_location[folderIndex]), (None, None, []))[2])
        negtoposaug = len(next(walk(neg2pos_aug_location[folderIndex]), (None, None, []))[2])
        negtoposreal = len(next(walk(neg2pos_real_location[folderIndex]), (None, None, []))[2])

        #print ('No of Files in TruePos-> copyFolder: ' + str(posaug) + ' ; applyFolder: '+ str(posreal))
        #print ('No of Files in TrueNeg-> copyFolder: ' + str(negaug) + ' ; applyFolder: '+ str(negreal))
        #print ('No of Files in PosToNeg-> copyFolder: ' + str(postonegaug) + ' ; applyFolder: '+ str(postonegreal))
        #print ('No of Files in NegToPos-> copyFolder: ' + str(negtoposaug) + ' ; applyFolder: '+ str(negtoposreal))

    # apply function to all folders
    r_processFolder = lambda x: processFolder (folderIndex=x)
    process_folders = list(map(r_processFolder, list(range(len(copyFolder)))))

    # Finish Job
    if verbose is True:
        print('Cloning Folder is complete, head over to /CSPOT/Thumbnails" to view results')