(ph HSrSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSK r SSK r SSK r SSK J r SSKJr SSKJr SSKJr SSKJr SSKJr SS KJr SSKrSS KJr SS KJr SS KJr S rSrSr SSSSSSS.r!Sr"Sr#Sr$0r%Sr&\&SSSSS\"\4Sj5r'\&SSSS\"\4SS.Sjj5r(\&SSSSSSSS\"\4 SS .S!jj5r)\&SSSSSS"S\"\4 SS.S#jj5r*\SS\"\4S"S$.S%jjr+S\S\"\4S&\\S'.S(jjr,S)r-S*r.S+r/"S,S-5r0S.r1S/r2S0r3S1r4S2r5"S3S45r6g)5zConcurrent media operations.N) exceptions)Client)Blob)_get_host_name)_quote)_DEFAULT_TIMEOUT) DEFAULT_RETRY)XMLMPUContainer) XMLMPUPart)DataCorruptionii@z Cache-ControlzContent-DispositionzContent-EncodingzContent-Languagezx-goog-custom-timezx-goog-storage-class) cacheControlcontentDispositioncontentEncodingcontentLanguage customTime storageClassprocessthreadzChecksum mismatch while downloading: {} The object metadata indicated a crc32c checksum of: {} but the actual crc32c checksum of the downloaded contents was: {} cF^[R"T5U4Sj5nU$)Nc>[R"T5R"U0UD6nURR S5nU(aURR S5nURR S5nU(dU(a [ S5e[ R"S5 URnURn[US'X1S'T"U0UD6$T"U0UD6$)Nthreads worker_type max_workerszuThe `threads` parameter is deprecated and conflicts with its replacement parameters, `worker_type` and `max_workers`.zeThe `threads` parameter is deprecated. Please use `worker_type` and `max_workers` parameters instead.) inspect signaturebind argumentsget ValueErrorwarningswarnargskwargsTHREAD)r#r$bindingrrrfuncs X/var/www/html/venv/lib/python3.13/site-packages/google/cloud/storage/transfer_manager.pyconvert_threads_or_raise:_deprecate_threads_param..convert_threads_or_raiseNs##D)..??##'' 2 !++// >K!++// >Kk L MMw <r?r@rArBrCrDrErG)blob_file_pairsdownload_kwargsrrKrLrrrIrMrNrOrBrQrPrRrSrTs r( download_manyr^s_H!3OI!A+!NJ  ,"1 Djs&C&C \*\3"?"?77>>,// NN6,:N4(!,4440 & #2.  :3E3E3S3S  3 -:G""$Cs#v}}' NO - ,s C?F F)additional_blob_attributesc  *Uc0nU c0n /n UHsn[RRX.5nX>-nUR"U40UD6nU R 5Hunn[ UUU5 M U R UU45 Mu [U UUUU U U S9$)aUpload many files concurrently by their filenames. The destination blobs are automatically created, with blob names based on the source filenames and the blob_name_prefix. For example, if the `filenames` include "images/icon.jpg", `source_directory` is "/home/myuser/", and `blob_name_prefix` is "myfiles/", then the file at "/home/myuser/images/icon.jpg" will be uploaded to a blob named "myfiles/images/icon.jpg". :type bucket: :class:`google.cloud.storage.bucket.Bucket` :param bucket: The bucket which will contain the uploaded blobs. :type filenames: list(str) :param filenames: A list of filenames to be uploaded. This may include part of the path. The file will be accessed at the full path of `source_directory` + `filename`. :type source_directory: str :param source_directory: A string that will be prepended (with `os.path.join()`) to each filename in the input list, in order to find the source file for each blob. Unlike the filename itself, the source_directory does not affect the name of the uploaded blob. For instance, if the source_directory is "/tmp/img/" and a filename is "0001.jpg", with an empty blob_name_prefix, then the file uploaded will be "/tmp/img/0001.jpg" and the destination blob will be "0001.jpg". This parameter can be an empty string. Note that this parameter allows directory traversal (e.g. "/", "../") and is not intended for unsanitized end user input. :type blob_name_prefix: str :param blob_name_prefix: A string that will be prepended to each filename in the input list, in order to determine the name of the destination blob. Unlike the filename itself, the prefix string does not affect the location the library will look for the source data on the local filesystem. For instance, if the source_directory is "/tmp/img/", the blob_name_prefix is "myuser/mystuff-" and a filename is "0001.jpg" then the file uploaded will be "/tmp/img/0001.jpg" and the destination blob will be "myuser/mystuff-0001.jpg". The blob_name_prefix can be blank (an empty string). :type skip_if_exists: bool :param skip_if_exists: If True, blobs that already have a live version will not be overwritten. This is accomplished by setting `if_generation_match = 0` on uploads. Uploads so skipped will result in a 412 Precondition Failed response code, which will be included in the return value, but not raised as an exception regardless of the value of raise_exception. :type blob_constructor_kwargs: dict :param blob_constructor_kwargs: A dictionary of keyword arguments to pass to the blob constructor. Refer to the documentation for `blob.Blob()` for more information. The dict is directly passed into the constructor and is not validated by this function. `name` and `bucket` keyword arguments are reserved by this function and will result in an error if passed in here. :type upload_kwargs: dict :param upload_kwargs: A dictionary of keyword arguments to pass to the upload method. Refer to the documentation for `blob.upload_from_file()` or `blob.upload_from_filename()` for more information. The dict is directly passed into the upload methods and is not validated by this function. :type threads: int :param threads: ***DEPRECATED*** Sets `worker_type` to THREAD and `max_workers` to the number specified. If `worker_type` or `max_workers` are set explicitly, this parameter should be set to None. Please use `worker_type` and `max_workers` instead of this parameter. :type deadline: int :param deadline: The number of seconds to wait for all threads to resolve. If the deadline is reached, all threads will be terminated regardless of their progress and `concurrent.futures.TimeoutError` will be raised. This can be left as the default of `None` (no deadline) for most use cases. :type raise_exception: bool :param raise_exception: If True, instead of adding exceptions to the list of return values, instead they will be raised. Note that encountering an exception on one operation will not prevent other operations from starting. Exceptions are only processed and potentially raised after all operations are complete in success or failure. If skip_if_exists is True, 412 Precondition Failed responses are considered part of normal operation and are not raised as an exception. :type worker_type: str :param worker_type: The worker type to use; one of `google.cloud.storage.transfer_manager.PROCESS` or `google.cloud.storage.transfer_manager.THREAD`. Although the exact performance impact depends on the use case, in most situations the PROCESS worker type will use more system resources (both memory and CPU) and result in faster operations than THREAD workers. Because the subprocesses of the PROCESS worker type can't access memory from the main process, Client objects have to be serialized and then recreated in each subprocess. The serialization of the Client object for use in subprocesses is an approximation and may not capture every detail of the Client object, especially if the Client was modified after its initial creation or if `Client._http` was modified in any way. THREAD worker types are observed to be relatively efficient for operations with many small files, but not for operations with large files. PROCESS workers are recommended for large file operations. :type max_workers: int :param max_workers: The maximum number of workers to create to handle the workload. With PROCESS workers, a larger number of workers will consume more system resources (memory and CPU) at once. How many workers is optimal depends heavily on the specific use case, and the default is a conservative number that should work okay in most cases without consuming excessive resources. :type additional_blob_attributes: dict :param additional_blob_attributes: A dictionary of blob attribute names and values. This allows the configuration of blobs beyond what is possible with blob_constructor_kwargs. For instance, {"cache_control": "no-cache"} would set the cache_control attribute of each blob to "no-cache". As with blob_constructor_kwargs, this affects the creation of every blob identically. To fine-tune each blob individually, use `upload_many` and create the blobs as desired before passing them in. :raises: :exc:`concurrent.futures.TimeoutError` if deadline is exceeded. :rtype: list :returns: A list of results corresponding to, in order, each item in the input list. If an exception was received, it will be the result for that operation. Otherwise, the return value from the successful upload method is used (which will be None). )rIrJrKrLrr)rYrZjoinrQitemssetattrr=rU)bucket filenamessource_directoryblob_name_prefixrIblob_constructor_kwargsrJrrKrLrrr`rHfilenamerZ blob_namerQpropvalues r(upload_many_from_filenamesrnsJ&"$!)%'"Oww||,7$/ {{9@(?@5;;=KD% D$ &>d|,  %#' r+Tc  </n UHn X=-n[RRX-5nU(a7[RRU5unn[R"USS9 U R UR U5U45 M [U UUUU U U S9$)a|Download many files concurrently by their blob names. The destination files are automatically created, with paths based on the source blob_names and the destination_directory. The destination files are not automatically deleted if their downloads fail, so please check the return value of this function for any exceptions, or enable `raise_exception=True`, and process the files accordingly. For example, if the `blob_names` include "icon.jpg", `destination_directory` is "/home/myuser/", and `blob_name_prefix` is "images/", then the blob named "images/icon.jpg" will be downloaded to a file named "/home/myuser/icon.jpg". :type bucket: :class:`google.cloud.storage.bucket.Bucket` :param bucket: The bucket which contains the blobs to be downloaded :type blob_names: list(str) :param blob_names: A list of blobs to be downloaded. The blob name in this string will be used to determine the destination file path as well. The full name to the blob must be blob_name_prefix + blob_name. The blob_name is separate from the blob_name_prefix because the blob_name will also determine the name of the destination blob. Any shared part of the blob names that need not be part of the destination path should be included in the blob_name_prefix. :type destination_directory: str :param destination_directory: A string that will be prepended (with os.path.join()) to each blob_name in the input list, in order to determine the destination path for that blob. For instance, if the destination_directory string is "/tmp/img" and a blob_name is "0001.jpg", with an empty blob_name_prefix, then the source blob "0001.jpg" will be downloaded to destination "/tmp/img/0001.jpg" . This parameter can be an empty string. Note that this parameter allows directory traversal (e.g. "/", "../") and is not intended for unsanitized end user input. :type blob_name_prefix: str :param blob_name_prefix: A string that will be prepended to each blob_name in the input list, in order to determine the name of the source blob. Unlike the blob_name itself, the prefix string does not affect the destination path on the local filesystem. For instance, if the destination_directory is "/tmp/img/", the blob_name_prefix is "myuser/mystuff-" and a blob_name is "0001.jpg" then the source blob "myuser/mystuff-0001.jpg" will be downloaded to "/tmp/img/0001.jpg". The blob_name_prefix can be blank (an empty string). :type download_kwargs: dict :param download_kwargs: A dictionary of keyword arguments to pass to the download method. Refer to the documentation for `blob.download_to_file()` or `blob.download_to_filename()` for more information. The dict is directly passed into the download methods and is not validated by this function. :type threads: int :param threads: ***DEPRECATED*** Sets `worker_type` to THREAD and `max_workers` to the number specified. If `worker_type` or `max_workers` are set explicitly, this parameter should be set to None. Please use `worker_type` and `max_workers` instead of this parameter. :type deadline: int :param deadline: The number of seconds to wait for all threads to resolve. If the deadline is reached, all threads will be terminated regardless of their progress and `concurrent.futures.TimeoutError` will be raised. This can be left as the default of `None` (no deadline) for most use cases. :type create_directories: bool :param create_directories: If True, recursively create any directories that do not exist. For instance, if downloading object "images/img001.png", create the directory "images" before downloading. :type raise_exception: bool :param raise_exception: If True, instead of adding exceptions to the list of return values, instead they will be raised. Note that encountering an exception on one operation will not prevent other operations from starting. Exceptions are only processed and potentially raised after all operations are complete in success or failure. If skip_if_exists is True, 412 Precondition Failed responses are considered part of normal operation and are not raised as an exception. :type worker_type: str :param worker_type: The worker type to use; one of `google.cloud.storage.transfer_manager.PROCESS` or `google.cloud.storage.transfer_manager.THREAD`. Although the exact performance impact depends on the use case, in most situations the PROCESS worker type will use more system resources (both memory and CPU) and result in faster operations than THREAD workers. Because the subprocesses of the PROCESS worker type can't access memory from the main process, Client objects have to be serialized and then recreated in each subprocess. The serialization of the Client object for use in subprocesses is an approximation and may not capture every detail of the Client object, especially if the Client was modified after its initial creation or if `Client._http` was modified in any way. THREAD worker types are observed to be relatively efficient for operations with many small files, but not for operations with large files. PROCESS workers are recommended for large file operations. :type max_workers: int :param max_workers: The maximum number of workers to create to handle the workload. With PROCESS workers, a larger number of workers will consume more system resources (memory and CPU) at once. How many workers is optimal depends heavily on the specific use case, and the default is a conservative number that should work okay in most cases without consuming excessive resources. :type skip_if_exists: bool :param skip_if_exists: Before downloading each blob, check if the file for the filename exists; if it does, skip that blob. This only works for filenames. :raises: :exc:`concurrent.futures.TimeoutError` if deadline is exceeded. :rtype: list :returns: A list of results corresponding to, in order, each item in the input list. If an exception was received, it will be the result for that operation. Otherwise, the return value from the successful download method is used (which will be None). T)exist_ok)r]rKrLrrrI)rYrZrbsplitmakedirsr=rQr^)re blob_namesdestination_directoryrhr]rrKcreate_directoriesrLrrrIr\rkfull_blob_namerZ directory_s r(download_many_to_pathryZspO )5ww||1= 77==.LIq KK D 1 N ;TBC   ''% r+)crc32c_checksumc URnUc0nSU;dSU;a [S5eSU;a [S5eUR5nSUS'SUS'UR(aUR(dUR 5 [ U5upU (a [U5OUn /n [US 5n SSS5 U "US 9nS nURnUU:aCUn[X-U5nU RUR[U UUUS - UUS 95 UU:aMC[RRX[RR S9 SSS5 /nU H"nURUR#55 M$ U(aU(a[%U5n[&R("U5R+S5nUR,nUU:wapUR/UUR1S5UR1S5UR1S5UR1S5S9n[3S[4R7UUU55eg!,(df  GN=f!,(df  GN =f)a"Download a single file in chunks, concurrently. In some environments, using this feature with mutiple processes will result in faster downloads of large files. Using this feature with multiple threads is unlikely to improve download performance under normal circumstances due to Python interpreter threading behavior. The default is therefore to use processes instead of threads. :type blob: :class:`google.cloud.storage.blob.Blob` :param blob: The blob to be downloaded. :type filename: str :param filename: The destination filename or path. :type chunk_size: int :param chunk_size: The size in bytes of each chunk to send. The optimal chunk size for maximum throughput may vary depending on the exact network environment and size of the blob. :type download_kwargs: dict :param download_kwargs: A dictionary of keyword arguments to pass to the download method. Refer to the documentation for `blob.download_to_file()` or `blob.download_to_filename()` for more information. The dict is directly passed into the download methods and is not validated by this function. Keyword arguments "start" and "end" which are not supported and will cause a ValueError if present. The key "checksum" is also not supported in `download_kwargs`, but see the argument `crc32c_checksum` (which does not go in `download_kwargs`) below. :type deadline: int :param deadline: The number of seconds to wait for all threads to resolve. If the deadline is reached, all threads will be terminated regardless of their progress and `concurrent.futures.TimeoutError` will be raised. This can be left as the default of `None` (no deadline) for most use cases. :type worker_type: str :param worker_type: The worker type to use; one of `google.cloud.storage.transfer_manager.PROCESS` or `google.cloud.storage.transfer_manager.THREAD`. Although the exact performance impact depends on the use case, in most situations the PROCESS worker type will use more system resources (both memory and CPU) and result in faster operations than THREAD workers. Because the subprocesses of the PROCESS worker type can't access memory from the main process, Client objects have to be serialized and then recreated in each subprocess. The serialization of the Client object for use in subprocesses is an approximation and may not capture every detail of the Client object, especially if the Client was modified after its initial creation or if `Client._http` was modified in any way. THREAD worker types are observed to be relatively efficient for operations with many small files, but not for operations with large files. PROCESS workers are recommended for large file operations. :type max_workers: int :param max_workers: The maximum number of workers to create to handle the workload. With PROCESS workers, a larger number of workers will consume more system resources (memory and CPU) at once. How many workers is optimal depends heavily on the specific use case, and the default is a conservative number that should work okay in most cases without consuming excessive resources. :type crc32c_checksum: bool :param crc32c_checksum: Whether to compute a checksum for the resulting object, using the crc32c algorithm. As the checksums for each chunk must be combined using a feature of crc32c that is not available for md5, md5 is not supported. :raises: :exc:`concurrent.futures.TimeoutError` if deadline is exceeded. :exc:`google.cloud.storage._media.common.DataCorruption` if the download's checksum doesn't agree with server-computed checksum. The `google.cloud.storage._media` exception is used here for consistency with other download methods despite the exception originating elsewhere. NstartendzWDownload arguments 'start' and 'end' are not supported by download_chunks_concurrently.checksumz'checksum' is in download_kwargs, but is not supported because sliced downloads have a different checksum mechanism from regular downloads. Use the 'crc32c_checksum' argument on download_chunks_concurrently instead.ztm.download_shardedr1wbr2r)r|r}r]rzr6zutf-8r0if_generation_not_matchif_metageneration_matchif_metageneration_not_match)r0rrr)clientr r9size generationreloadr:r@openminr=r>"_download_and_write_chunk_in_placerArBrCrDrG'_digest_ordered_checksum_and_size_pairsbase64 b64encodedecodecrc32c_get_download_urlrr !DOWNLOAD_CRC32C_MISMATCH_TEMPLATEformat)rQrj chunk_sizer]rKrrrzrrMrNmaybe_pickled_blobrBrxrOcursorr}r|rRrS crc_digestactual_checksumexpected_checksum download_urls r(download_chunks_concurrentlyrsgF[[F/!Uo%= e  _$ f  &**,O"&OJ!6OI 99DOO !A+!NJ1?-TG h     ,iislE,c2F NN6& $3$3  sl  :3E3E3S3S  % -.Gv}}'7 _upload_partrArBrCrDrG register_partfinalize Exceptioncancel) rjrQrrrKrrr~r7rrerrrurl base_headersobject_metadatar containerrr num_of_partsrMrNmaybe_pickled_clientrBrOrr|r}rSetags r(upload_chunks_concurrentlyrs|[[F [[F##F+If001H & - -++F4994E . C372L2Lx9L3M3/L/<JI!7!HIG $)-):):%& $)&1G  , L1$45K 1_ 2EkJ.5C NN ( +%#   6(  X:3E3E3S3S  + -2 F &  K  # #K 6 4..v67? - ,@ ,,V45 sBH7(AI7 I,I4c [U[5(aUn O[R"U5n [ UUUUUUUUU S9 n U R U R 5 XkR4$)zHelper function that runs inside a thread or subprocess to upload a part. `maybe_pickled_client` is either a Client (for threads) or a specially pickled Client (for processes) because the default pickling mangles Client objects.r)r;rpickleloadsr upload_httpr) rrrrjr|r}rr~rrrparts r(rrsj$&//%23    D KK   ##r+c0nUR5Hup#U[;dMX1[U'M SU;a#USR5H up#X1SU-'M U$)zFHelper function to translate object metadata into a header dictionary.metadataz x-goog-meta-)rcMETADATA_HEADER_TRANSLATION)rrkeyrms r(rrsiGnn&  - -8=/4 5'X":.446JC,1NS( )7 Nr+c[U[5(aUnO[R"U5n[ XU5nUR "U4X#S.UD6 UR X2- S-4sSSS5 $!,(df  g=f)a(Helper function that runs inside a thread or subprocess. `maybe_pickled_blob` is either a Blob (for threads) or a specially pickled Blob (for processes) because the default pickling mangles Client objects which are attached to Blobs. Returns a crc if configured (or None) and the size written. )r|r}rN)r;rrr_ChecksummingSparseFileWrapperrXcrc)rrjr|r}r]rzrQfs r(rrsj$d++!||./ ' IQ ""1NENoN q() J I Is (A-- A;c@\rSrSrSrSrSr\S5rSr Sr Sr g ) rizA file wrapper that writes to a sparse file and optionally checksums. This wrapper only implements write() and does not inherit from `io` module base classes. cv[US5UlURRU5 SUlX0lg)Nzrb+)rrseek_crc_crc32c_enabled)selfrjstart_positioncrc32c_enableds r(__init__'_ChecksummingSparseFileWrapper.__init__s-h&  N# -r+cUR(aOURc[R"U5UlO&[R"URU5UlUR R U5 gN)rr google_crc32crmextendrwrite)rchunks r(r$_ChecksummingSparseFileWrapper.writesO   yy )//6 )00EB   Ur+cUR$r)rrs r(r"_ChecksummingSparseFileWrapper.crcs yyr+cU$rrs r( __enter__(_ChecksummingSparseFileWrapper.__enter__s r+c8URR5 gr)rclose)rexc_type exc_valuetbs r(__exit__'_ChecksummingSparseFileWrapper.__exit__s  r+)rrrN) __name__ __module__ __qualname____firstlineno____doc__rrpropertyrrr__static_attributes__rr+r(rrs/ .r+rc[U[5(aUnO[R"U5n[ XA5"U0UD6$)zHelper function that runs inside a thread or subprocess. `maybe_pickled_blob` is either a Blob (for threads) or a specially pickled Blob (for processes) because the default pickling mangles Client objects which are attached to Blobs.)r;rrrgetattr)r method_namer#r$rQs r(r?r?s;$d++!||./ 4 %t 6v 66r+c[U5nURnURnSnURnURnUR n[ UUUUUUU44$)zReplicate a Client by constructing a new one with the same params. LazyClient performs transparent caching for when the same client is needed on the same process multiple times.N)idproject _credentials_initial_client_info_initial_client_options_extra_headers _LazyClient)clclient_object_idr credentialsr client_infoclient_options extra_headerss r(_reduce_clientrsm "vjjG//K E))K//N%%M   r+c[R"5n[R"U5n[R R 5Ul[UR ['URU5 UR5$)z=Pickle a Client or an object that owns a Client (like a Blob)) ioBytesIOrPicklercopyregdispatch_tabler9rrdumpgetvalue)objrps r(r@r@s[ AqA--224A-AVFF3K ::<r+cU[:Xa[RRS4$U[:Xa[RR S4$[ S5e)zDReturns the pool class, and whether the pool requires pickled Blobs.TFzuThe worker_type must be google.cloud.storage.transfer_manager.PROCESS or google.cloud.storage.transfer_manager.THREAD)PROCESSrArBProcessPoolExecutorr%ThreadPoolExecutorr )rs r(r:r:-sUg""66==  ""55u== D  r+cSn[[5nUH[up4U(dUnMUS-nSnXT:a6[XE- [5n[R"XSU5nXV- nXT:aM6US-nX-nM] [ R "SU5nU$)Nlrz>L)bytesMAX_CRC32C_ZERO_ARRAY_SIZErrrstructpack)checksum_and_size_pairsbase_crczeroespart_crcrpaddeddesired_zeroes_sizers r(rr<sH - .F1H  "H F-&)4=;U&V#(//AUBU:VW--  "H  H!2" hJ r+c\rSrSrSrSrSrg)riVzBAn object that will transform into either a cached or a new Clientcj[RU5nU(aU$[U0UD6nU[U'U$r)_cached_clientsrr)clsrr#r$ cached_clients r(__new___LazyClient.__new__Ys:'++B/  "D3F3M"/OB  r+rN)rrrrrr!rrr+r(rrVs L!r+r)7rconcurrent.futuresrArrrYr!rrrrr,google.api_corergoogle.cloud.storagerrgoogle.cloud.storage.blobrrgoogle.cloud.storage.constantsrgoogle.cloud.storage.retryr r+google.cloud.storage._media.requests.uploadr r google.cloud.storage.exceptionsr TM_DEFAULT_CHUNK_SIZEDEFAULT_MAX_WORKERSrrrr%rrr.rUr^rnryrrrrrrr?rr@r:rrrr+r(r-s#    &'%4,;4GB:(,#/))&*    %!$6  #ZZz  #QQQh   #{ ${{|  #iii^% #ttt$ #G  GT"$J *,B 7 2"  4 ! !r+