File: //opt/cppython/lib/python3.8/site-packages/s3transfer/__pycache__/upload.cpython-38.pyc
U
LD�g
| � @ s� d dl Z d dlmZ d dlmZmZ d dlmZ d dlm Z d dl
m Z m Z m
Z
mZ d dlmZmZmZmZ G dd � d �ZG d
d � d �ZG d d
� d
�ZG dd� de�ZG dd� de�ZG dd� de�ZG dd� de
�ZG dd� de�ZG dd� de�Z dS )� N)�BytesIO��readable�seekable)�FULL_OBJECT_CHECKSUM_ARGS)�IN_MEMORY_UPLOAD_TAG)�CompleteMultipartUploadTask�CreateMultipartUploadTask�SubmissionTask�Task)�ChunksizeAdjuster�DeferredOpenFile�
get_callbacks�get_filtered_dictc @ s. e Zd Zd dd�Zdd� Zdd� Zdd � Zd
S ) �AggregatedProgressCallback� c C s || _ || _d| _dS )a� Aggregates progress updates for every provided progress callback
:type callbacks: A list of functions that accepts bytes_transferred
as a single argument
:param callbacks: The callbacks to invoke when threshold is reached
:type threshold: int
:param threshold: The progress threshold in which to take the
aggregated progress and invoke the progress callback with that
aggregated progress total
r N)�
_callbacks�
_threshold� _bytes_seen)�self� callbacks� threshold� r �>/opt/cppython/lib/python3.8/site-packages/s3transfer/upload.py�__init__" s z#AggregatedProgressCallback.__init__c C s& | j |7 _ | j | jkr"| �� d S �N)r r �_trigger_callbacks)r �bytes_transferredr r r �__call__2 s z#AggregatedProgressCallback.__call__c C s | j dkr| �� dS )z@Flushes out any progress that has not been sent to its callbacksr N)r r �r r r r �flush7 s
z AggregatedProgressCallback.flushc C s" | j D ]}|| jd� qd| _d S )N)r r )r r )r �callbackr r r r < s
z-AggregatedProgressCallback._trigger_callbacksN)r )�__name__�
__module__� __qualname__r r r r r r r r r ! s
r c @ sL e Zd ZdZdd� Zddd�Zddd �Zd
d � Zd d
� Zdd� Z dd� Z
dS )�InterruptReadera� Wrapper that can interrupt reading using an error
It uses a transfer coordinator to propagate an error if it notices
that a read is being made while the file is being read from.
:type fileobj: file-like obj
:param fileobj: The file-like object to read from
:type transfer_coordinator: s3transfer.futures.TransferCoordinator
:param transfer_coordinator: The transfer coordinator to use if the
reader needs to be interrupted.
c C s || _ || _d S r )�_fileobj�_transfer_coordinator)r �fileobj�transfer_coordinatorr r r r P s zInterruptReader.__init__Nc C s | j jr| j j�| j�|�S r )r' � exceptionr& �read)r �amountr r r r+ T s zInterruptReader.readr c C s | j �||� d S r )r& �seek)r �where�whencer r r r- ^ s zInterruptReader.seekc C s
| j �� S r )r& �tellr r r r r0 a s zInterruptReader.tellc C s | j �� d S r )r& �closer r r r r1 d s zInterruptReader.closec C s | S r r r r r r � __enter__g s zInterruptReader.__enter__c O s | � � d S r )r1 )r �args�kwargsr r r �__exit__j s zInterruptReader.__exit__)N)r ) r" r# r$ �__doc__r r+ r- r0 r1 r2 r5 r r r r r% B s
r% c @ sf e Zd ZdZddd�Zedd� �Zdd� Zd d
� Zd d � Z d
d� Z
dd� Z dd� Z dd� Z
dd� ZdS )�UploadInputManageraJ Base manager class for handling various types of files for uploads
This class is typically used for the UploadSubmissionTask class to help
determine the following:
* How to determine the size of the file
* How to determine if a multipart upload is required
* How to retrieve the body for a PutObject
* How to retrieve the bodies for a set of UploadParts
The answers/implementations differ for the various types of file inputs
that may be accepted. All implementations must subclass and override
public methods from this class.
Nc C s || _ || _|| _d S r )�_osutilr' �_bandwidth_limiter�r �osutilr) �bandwidth_limiterr r r r ~ s zUploadInputManager.__init__c C s t d��dS )a Determines if the source for the upload is compatible with manager
:param upload_source: The source for which the upload will pull data
from.
:returns: True if the manager can handle the type of source specified
otherwise returns False.
z must implement _is_compatible()N��NotImplementedError��clsZ
upload_sourcer r r �
is_compatible� s
z UploadInputManager.is_compatiblec C s t d��dS )a� Whether the body it provides are stored in-memory
:type operation_name: str
:param operation_name: The name of the client operation that the body
is being used for. Valid operation_names are ``put_object`` and
``upload_part``.
:rtype: boolean
:returns: True if the body returned by the manager will be stored in
memory. False if the manager will not directly store the body in
memory.
z%must implement store_body_in_memory()Nr= �r �operation_namer r r �stores_body_in_memory� s
z(UploadInputManager.stores_body_in_memoryc C s t d��dS )z�Provides the transfer size of an upload
:type transfer_future: s3transfer.futures.TransferFuture
:param transfer_future: The future associated with upload request
z&must implement provide_transfer_size()Nr= �r �transfer_futurer r r �provide_transfer_size� s z(UploadInputManager.provide_transfer_sizec C s t d��dS )a� Determines where a multipart upload is required
:type transfer_future: s3transfer.futures.TransferFuture
:param transfer_future: The future associated with upload request
:type config: s3transfer.manager.TransferConfig
:param config: The config associated to the transfer manager
:rtype: boolean
:returns: True, if the upload should be multipart based on
configuration and size. False, otherwise.
z*must implement requires_multipart_upload()Nr= �r rF �configr r r �requires_multipart_upload� s
z,UploadInputManager.requires_multipart_uploadc C s t d��dS )a� Returns the body to use for PutObject
:type transfer_future: s3transfer.futures.TransferFuture
:param transfer_future: The future associated with upload request
:type config: s3transfer.manager.TransferConfig
:param config: The config associated to the transfer manager
:rtype: s3transfer.utils.ReadFileChunk
:returns: A ReadFileChunk including all progress callbacks
associated with the transfer future.
z$must implement get_put_object_body()Nr= rE r r r �get_put_object_body� s
z&UploadInputManager.get_put_object_bodyc C s t d��dS )a Yields the part number and body to use for each UploadPart
:type transfer_future: s3transfer.futures.TransferFuture
:param transfer_future: The future associated with upload request
:type chunksize: int
:param chunksize: The chunksize to use for this upload.
:rtype: int, s3transfer.utils.ReadFileChunk
:returns: Yields the part number and the ReadFileChunk including all
progress callbacks associated with the transfer future for that
specific yielded part.
z)must implement yield_upload_part_bodies()Nr= )r rF � chunksizer r r �yield_upload_part_bodies� s z+UploadInputManager.yield_upload_part_bodiesc C s* t || j�}| jr&| jj|| jdd�}|S )NF)�enabled)r% r' r9 Zget_bandwith_limited_stream)r r( r r r �
_wrap_fileobj� s �z UploadInputManager._wrap_fileobjc C s t |d�}|rt|�gS g S )N�progress)r r )r rF r r r r �_get_progress_callbacks� s
z*UploadInputManager._get_progress_callbacksc C s dd� |D �S )Nc S s g | ]
}|j �qS r )r )�.0r! r r r �
<listcomp>� s z;UploadInputManager._get_close_callbacks.<locals>.<listcomp>r )r Z aggregated_progress_callbacksr r r �_get_close_callbacks� s z'UploadInputManager._get_close_callbacks)N)r" r# r$ r6 r � classmethodrA rD rG rJ rK rM rO rQ rT r r r r r7 n s
r7 c @ sd e Zd ZdZedd� �Zdd� Zdd� Zdd � Zd
d � Z d d
� Z
dd� Z dd� Z dd� Z
dd� ZdS )�UploadFilenameInputManagerz Upload utility for filenamesc C s
t |t�S r )�
isinstance�strr? r r r rA � s z(UploadFilenameInputManager.is_compatiblec C s dS )NFr rB r r r rD � s z0UploadFilenameInputManager.stores_body_in_memoryc C s |j �| j�|j jj�� d S r )�metarG r8 Z
get_file_size� call_argsr( rE r r r rG � s �z0UploadFilenameInputManager.provide_transfer_sizec C s |j j|jkS r )rY �size�multipart_thresholdrH r r r rJ � s z4UploadFilenameInputManager.requires_multipart_uploadc C sJ | � |�\}}| �|�}| �|�}| �|�}|jj}| jj|||||d�S )N�r( �
chunk_size�full_file_sizer �close_callbacks)�&_get_put_object_fileobj_with_full_sizerO rQ rT rY r[ r8 �#open_file_chunk_reader_from_fileobj)r rF r( � full_sizer r` r[ r r r rK � s �
�z.UploadFilenameInputManager.get_put_object_bodyc c s� |j j}| �||�}td|d �D ]j}| �|�}| �|�}||d }| j|j jj|||d�\} }
| � | �} | j
j | ||
||d�} || fV q"d S )N� )�
start_byte� part_sizer_ r] ) rY r[ �_get_num_parts�rangerQ rT �'_get_upload_part_fileobj_with_full_sizerZ r( rO r8 rb ) r rF rL r_ Z num_parts� part_numberr r` re r( rc Zread_file_chunkr r r rM s*
�
�z3UploadFilenameInputManager.yield_upload_part_bodiesc C s t ||| jjd�}|S )N)Z
open_function)r
r8 �open)r r( re r r r �_get_deferred_open_file2 s �z2UploadFilenameInputManager._get_deferred_open_filec C s" |j jj}|j j}| �|d�|fS )Nr )rY rZ r( r[ rl �r rF r( r[ r r r ra 8 s
zAUploadFilenameInputManager._get_put_object_fileobj_with_full_sizec K s |d }|d }| � ||�|fS )Nre r_ )rl )r r( r4 re rc r r r ri = s zBUploadFilenameInputManager._get_upload_part_fileobj_with_full_sizec C s t t�|jjt|� ��S r )�int�math�ceilrY r[ �float)r rF rf r r r rg B s z)UploadFilenameInputManager._get_num_partsN)r" r# r$ r6 rU rA rD rG rJ rK rM rl ra ri rg r r r r rV � s
rV c @ s< e Zd ZdZedd� �Zdd� Zdd� Zdd � Zd
d � Z d S )
�UploadSeekableInputManagerz&Upload utility for an open file objectc C s t |�ot|�S r r r? r r r rA I s z(UploadSeekableInputManager.is_compatiblec C s |dkr dS dS d S )N�
put_objectFTr rB r r r rD M s z0UploadSeekableInputManager.stores_body_in_memoryc C sD |j jj}|�� }|�dd� |�� }|�|� |j �|| � d S )Nr � )rY rZ r( r0 r- rG )r rF r( Zstart_positionZ end_positionr r r rG S s
�z0UploadSeekableInputManager.provide_transfer_sizec K s |� |d �}t|�t|�fS )Nrf )r+ r �len)r r( r4 �datar r r ri ` s zBUploadSeekableInputManager._get_upload_part_fileobj_with_full_sizec C s" |j jj}|�� |j j }||fS r )rY rZ r( r0 r[ rm r r r ra o s
zAUploadSeekableInputManager._get_put_object_fileobj_with_full_sizeN)
r" r# r$ r6 rU rA rD rG ri ra r r r r rr F s
rr c sh e Zd ZdZd� fdd� Zedd� �Zdd� Zd d
� Zd d � Z d
d� Z
dd� Z ddd�Z dd� Z
� ZS )� UploadNonSeekableInputManagerz7Upload utility for a file-like object that cannot seek.Nc s t � �|||� d| _d S )N� )�superr �
_initial_datar: �� __class__r r r z s z&UploadNonSeekableInputManager.__init__c C s t |�S r )r r? r r r rA ~ s z+UploadNonSeekableInputManager.is_compatiblec C s dS )NTr rB r r r rD � s z3UploadNonSeekableInputManager.stores_body_in_memoryc C s d S r r rE r r r rG � s z3UploadNonSeekableInputManager.provide_transfer_sizec C sT |j jd k r|j j|jkS |j jj}|j}| �||d�| _t| j�|k rLdS dS d S )NFT)rY r[ r\ rZ r( �_readrz ru )r rF rI r( r r r r rJ � s
z7UploadNonSeekableInputManager.requires_multipart_uploadc C s@ | � |�}| �|�}|jjj}| �| j|�� ||�}d | _|S r )rQ rT rY rZ r( �
_wrap_datarz r+ )r rF r r` r( �bodyr r r rK � s
�z1UploadNonSeekableInputManager.get_put_object_bodyc c s^ |j jj}d}| �|�}| �|�}|d7 }| �||�}|s<qZ| �|||�}d }||fV qd S )Nr rd )rY rZ r( rQ rT r} r~ ) r rF rL Z file_objectrj r r` Z part_contentZ part_objectr r r rM � s
�z6UploadNonSeekableInputManager.yield_upload_part_bodiesTc C sx t | j�dkr|�|�S |t | j�krL| jd|� }|rH| j|d� | _|S |t | j� }| j|�|� }|rtd| _|S )a=
Reads a specific amount of data from a stream and returns it. If there
is any data in initial_data, that will be popped out first.
:type fileobj: A file-like object that implements read
:param fileobj: The stream to read from.
:type amount: int
:param amount: The number of bytes to read from the stream.
:type truncate: bool
:param truncate: Whether or not to truncate initial_data after
reading from it.
:return: Generator which generates part bodies from the initial data.
r Nrx )ru rz r+ )r r( r, �truncaterv Zamount_to_readr r r r} � s
z#UploadNonSeekableInputManager._readc C s, | � t|��}| jj|t|�t|�||d�S )a�
Wraps data with the interrupt reader and the file chunk reader.
:type data: bytes
:param data: The data to wrap.
:type callbacks: list
:param callbacks: The callbacks associated with the transfer future.
:type close_callbacks: list
:param close_callbacks: The callbacks to be called when closing the
wrapper for the data.
:return: Fully wrapped data.
r] )rO r r8 rb ru )r rv r r` r( r r r r~ � s �z(UploadNonSeekableInputManager._wrap_data)N)T)r" r# r$ r6 r rU rA rD rG rJ rK rM r} r~ �
__classcell__r r r{ r rw w s
*rw c @ s� e Zd ZdZddgZedg Zdddddd gZddddd ddge Zd
d � Z d d
d�Z
dd� Z dd� Z dd� Z
dd� Zdd� Zdd� Zdd � Zd S ) �UploadSubmissionTaskz.Task for submitting tasks to execute an upload� ChecksumTypeZ
MpuObjectSize�ChecksumAlgorithmZSSECustomerKeyZSSECustomerAlgorithmZSSECustomerKeyMD5Z RequestPayerZExpectedBucketOwnerc C sN t ttg}|jjj}|D ]}|�|�r| S qtd|� dt|�� d���dS )ao Retrieves a class for managing input for an upload based on file type
:type transfer_future: s3transfer.futures.TransferFuture
:param transfer_future: The transfer future for the request
:rtype: class of UploadInputManager
:returns: The appropriate class to use for managing a specific type of
input for uploads.
zInput z
of type: z is not supported.N) rV rr rw rY rZ r( rA � RuntimeError�type)r rF Z upload_manager_resolver_chainr( Zupload_manager_clsr r r � _get_upload_input_manager_cls s �
�z2UploadSubmissionTask._get_upload_input_manager_clsNc C sd | � |�|| j|�}|jjdkr*|�|� |�||�sL| �||||||� n| �||||||� dS )a�
:param client: The client associated with the transfer manager
:type config: s3transfer.manager.TransferConfig
:param config: The transfer config associated with the transfer
manager
:type osutil: s3transfer.utils.OSUtil
:param osutil: The os utility associated to the transfer manager
:type request_executor: s3transfer.futures.BoundedExecutor
:param request_executor: The request executor associated with the
transfer manager
:type transfer_future: s3transfer.futures.TransferFuture
:param transfer_future: The transfer future associated with the
transfer request that tasks are being submitted for
N)r� r' rY r[ rG rJ �_submit_upload_request�_submit_multipart_request)r �clientrI r; �request_executorrF r< �upload_input_managerr r r �_submit3 s: � �
�� �z UploadSubmissionTask._submitc
C sX |j j}| �|j�}| �|d�} | jj|t| j||�|�|j |j
|d�dd�| d� d S )Nrs )r� r( �bucket�key�
extra_argsT)r) � main_kwargs�is_final��tag) rY rZ �_extra_put_object_argsr� �_get_upload_task_tagr' �submit�
PutObjectTaskrK r� r� )
r r� rI r; r� rF r� rZ Zput_object_extra_argsZput_object_tagr r r r� l s0 � ��� �
�z+UploadSubmissionTask._submit_upload_requestc C s6 |j j}tD ]*}||jkr d|jd< |�dd�|jd< q | �|j�} | j�|t| j||j |j
| d�d��}
g } | � |j�} | � |d�}
|j j
}t� }|�|j|�}|�||�}|D ]B\}}| �| jj|t| j|||j |j
|| d �d
|
id �|
d �� q�| �|j�}| j�|t| j||j |j
|d�|
| d
�dd�� d S )NZ FULL_OBJECTr� �Checksum� r� )r� r� r� r� )r) r� � upload_part)r� r( r� r� rj r� � upload_id)r) r� �pending_main_kwargsr� )r� �partsT)r) r� r� r� )rY rZ r r� �replace� _extra_create_multipart_argsr' r� r r� r� �_extra_upload_part_argsr� r[ r Zadjust_chunksizeZmultipart_chunksizerM �append�UploadPartTask� _extra_complete_multipart_argsr )r r� rI r; r� rF r� rZ ZchecksumZcreate_multipart_extra_argsZcreate_multipart_futureZ part_futuresZextra_part_argsZupload_part_tagr[ ZadjusterrL Z
part_iteratorrj r( Z complete_multipart_extra_argsr r r r� � s�
� ���� � � � ���������z.UploadSubmissionTask._submit_multipart_requestc C s t || j�S r )r �UPLOAD_PART_ARGS�r r� r r r r� � s z,UploadSubmissionTask._extra_upload_part_argsc C s t || j�S r )r �COMPLETE_MULTIPART_ARGSr� r r r r� � s z3UploadSubmissionTask._extra_complete_multipart_argsc C s t || jd�S �N)Zblocklisted_keys)r �CREATE_MULTIPART_BLOCKLISTr� r r r r� � s �z1UploadSubmissionTask._extra_create_multipart_argsc C s t || jd�S r� )r �PUT_OBJECT_BLOCKLISTr� r r r r� s �z+UploadSubmissionTask._extra_put_object_argsc C s d }|� |�rt}|S r )rD r )r r� rC r� r r r r� s
z)UploadSubmissionTask._get_upload_task_tag)N)r" r# r$ r6 r� r r� r� r� r� r� r� r� r� r� r� r� r� r r r r r� s<
�
��
�
9'cr� c @ s e Zd ZdZdd� ZdS )r� z Task to do a nonmultipart uploadc C s, |� }|j f |||d�|�� W 5 Q R X dS )aP
:param client: The client to use when calling PutObject
:param fileobj: The file to upload.
:param bucket: The name of the bucket to upload to
:param key: The name of the key to upload to
:param extra_args: A dictionary of any extra arguments that may be
used in the upload.
)�Bucket�Key�BodyN)rs )r r� r( r� r� r� r r r r �_main s zPutObjectTask._mainN�r" r# r$ r6 r� r r r r r� s r� c @ s e Zd ZdZdd� ZdS )r� z+Task to upload a part in a multipart uploadc C st |�"}|j f |||||d�|��} W 5 Q R X | d }
|
|d�} d|krp|d �� } d| � �}
|
| krp| |
| |
< | S )a�
:param client: The client to use when calling PutObject
:param fileobj: The file to upload.
:param bucket: The name of the bucket to upload to
:param key: The name of the key to upload to
:param upload_id: The id of the upload
:param part_number: The number representing the part of the multipart
upload
:param extra_args: A dictionary of any extra arguments that may be
used in the upload.
:rtype: dict
:returns: A dictionary representing a part::
{'Etag': etag_value, 'PartNumber': part_number}
This value can be appended to a list to be used to complete
the multipart upload.
)r� r� ZUploadId�
PartNumberr� �ETag)r� r� r� r� )r� �upper)r r� r( r� r� r� rj r� r �response�etagZ
part_metadataZalgorithm_nameZchecksum_memberr r r r� " s$ ��
zUploadPartTask._mainNr� r r r r r� s r� ) ro �ior Zs3transfer.compatr r Zs3transfer.constantsr Zs3transfer.futuresr Zs3transfer.tasksr r r
r Zs3transfer.utilsr r
r r r r% r7 rV rr rw r� r� r� r r r r �<module>
s$ !,{]1