File: //opt/cppython/lib/python3.8/site-packages/s3transfer/__pycache__/__init__.cpython-38.pyc
U
LD�g�s � @ sp d Z ddlZddlZddlZddlZddlZddlZddlZddl Z ddl
Z
ddl Z ddl m
Z
ddlmZmZ ddlmZ ddlZddlmZmZ dZdZG d d
� d
ej�Ze� e �Z e � e� � d Z e!� Z"d%d
d�Z#dd� Z$dd� Z%G dd� de&�Z'G dd� d�Z(G dd� d�Z)G dd� d�Z*G dd � d �Z+G d d � d ej,�Z-G d d � d �Z.G d!d"� d"�Z/G d#d$� d$�Z0dS )&a� Abstractions over S3's upload/download operations.
This module provides high level abstractions for efficient
uploads/downloads. It handles several things for the user:
* Automatically switching to multipart transfers when
a file is over a specific size threshold
* Uploading/downloading a file in parallel
* Throttling based on max bandwidth
* Progress callbacks to monitor transfers
* Retries. While botocore handles retries for streaming uploads,
it is not possible for it to handle retries for streaming
downloads. This module handles retries for both cases so
you don't need to implement any retry logic yourself.
This module has a reasonable set of defaults. It also allows you
to configure many aspects of the transfer process including:
* Multipart threshold size
* Max parallel downloads
* Max bandwidth
* Socket timeouts
* Retry amounts
There is no support for s3->s3 multipart copies at this
time.
.. _ref_s3transfer_usage:
Usage
=====
The simplest way to use this module is:
.. code-block:: python
client = boto3.client('s3', 'us-west-2')
transfer = S3Transfer(client)
# Upload /tmp/myfile to s3://bucket/key
transfer.upload_file('/tmp/myfile', 'bucket', 'key')
# Download s3://bucket/key to /tmp/myfile
transfer.download_file('bucket', 'key', '/tmp/myfile')
The ``upload_file`` and ``download_file`` methods also accept
``**kwargs``, which will be forwarded through to the corresponding
client operation. Here are a few examples using ``upload_file``::
# Making the object public
transfer.upload_file('/tmp/myfile', 'bucket', 'key',
extra_args={'ACL': 'public-read'})
# Setting metadata
transfer.upload_file('/tmp/myfile', 'bucket', 'key',
extra_args={'Metadata': {'a': 'b', 'c': 'd'}})
# Setting content type
transfer.upload_file('/tmp/myfile.json', 'bucket', 'key',
extra_args={'ContentType': "application/json"})
The ``S3Transfer`` class also supports progress callbacks so you can
provide transfer progress to users. Both the ``upload_file`` and
``download_file`` methods take an optional ``callback`` parameter.
Here's an example of how to print a simple progress percentage
to the user:
.. code-block:: python
class ProgressPercentage(object):
def __init__(self, filename):
self._filename = filename
self._size = float(os.path.getsize(filename))
self._seen_so_far = 0
self._lock = threading.Lock()
def __call__(self, bytes_amount):
# To simplify we'll assume this is hooked up
# to a single filename.
with self._lock:
self._seen_so_far += bytes_amount
percentage = (self._seen_so_far / self._size) * 100
sys.stdout.write(
"
%s %s / %s (%.2f%%)" % (self._filename, self._seen_so_far,
self._size, percentage))
sys.stdout.flush()
transfer = S3Transfer(boto3.client('s3', 'us-west-2'))
# Upload /tmp/myfile to s3://bucket/key and print upload progress.
transfer.upload_file('/tmp/myfile', 'bucket', 'key',
callback=ProgressPercentage('/tmp/myfile'))
You can also provide a TransferConfig object to the S3Transfer
object that gives you more fine grained control over the
transfer. For example:
.. code-block:: python
client = boto3.client('s3', 'us-west-2')
config = TransferConfig(
multipart_threshold=8 * 1024 * 1024,
max_concurrency=10,
num_download_attempts=10,
)
transfer = S3Transfer(client, config)
transfer.upload_file('/tmp/foo', 'bucket', 'key')
� N)�six)�IncompleteReadError�ResponseStreamingError)�ReadTimeoutError)�RetriesExceededError�S3UploadFailedErrorzAmazon Web Servicesz0.11.4c @ s e Zd Zdd� ZdS )� NullHandlerc C s d S �N� )�self�recordr
r
�@/opt/cppython/lib/python3.8/site-packages/s3transfer/__init__.py�emit� s zNullHandler.emitN)�__name__�
__module__� __qualname__r r
r
r
r
r � s r i � c C s d� dd� t| �D ��S )N� c s s | ]}t �tj�V qd S r )�random�choice�string� hexdigits)�.0�_r
r
r
� <genexpr>� s z(random_file_extension.<locals>.<genexpr>)�join�range)Z
num_digitsr
r
r
�random_file_extension� s r c K s" |dkr t | jd�r | j�� d S )N�Z PutObjectZ
UploadPart�disable_callback)�hasattr�bodyr ��requestZoperation_name�kwargsr
r
r
�disable_upload_callbacks� s
�r% c K s" |dkr t | jd�r | j�� d S )Nr �enable_callback)r r! r& r" r
r
r
�enable_upload_callbacks� s
�r' c @ s e Zd ZdS )�QueueShutdownErrorN)r r r r
r
r
r
r( � s r( c @ s~ e Zd Zd dd�Zed dd��Zdd� Zd d d
�Zd d � Zd
d� Z dd� Z
dd� Z dd� Z dd� Z
dd� Zdd� Zdd � ZdS ) �
ReadFileChunkNTc C sF || _ || _| j| j |||d�| _| j �| j� d| _|| _|| _dS )a�
Given a file object shown below:
|___________________________________________________|
0 | | full_file_size
|----chunk_size---|
start_byte
:type fileobj: file
:param fileobj: File like object
:type start_byte: int
:param start_byte: The first byte from which to start reading.
:type chunk_size: int
:param chunk_size: The max chunk size to read. Trying to read
pass the end of the chunk size will behave like you've
reached the end of the file.
:type full_file_size: int
:param full_file_size: The entire content length associated
with ``fileobj``.
:type callback: function(amount_read)
:param callback: Called whenever data is read from this object.
)�requested_size�
start_byte�actual_file_sizer N)�_fileobj� _start_byte�_calculate_file_size�_size�seek� _amount_read� _callback�_callback_enabled)r �fileobjr+ �
chunk_sizeZfull_file_size�callbackr& r
r
r
�__init__� s %�zReadFileChunk.__init__c C s, t |d�}t�|�� �j}| ||||||�S )aW Convenience factory function to create from a filename.
:type start_byte: int
:param start_byte: The first byte from which to start reading.
:type chunk_size: int
:param chunk_size: The max chunk size to read. Trying to read
pass the end of the chunk size will behave like you've
reached the end of the file.
:type full_file_size: int
:param full_file_size: The entire content length associated
with ``fileobj``.
:type callback: function(amount_read)
:param callback: Called whenever data is read from this object.
:type enable_callback: bool
:param enable_callback: Indicate whether to invoke callback
during read() calls.
:rtype: ``ReadFileChunk``
:return: A new instance of ``ReadFileChunk``
�rb)�open�os�fstat�fileno�st_size)�cls�filenamer+ r6 r7 r& �f� file_sizer
r
r
�
from_filename� s "
�zReadFileChunk.from_filenamec C s || }t ||�S r )�min)r r5 r* r+ r, Zmax_chunk_sizer
r
r
r/ s z"ReadFileChunk._calculate_file_sizec C sh |d kr| j | j }nt| j | j |�}| j�|�}| jt|�7 _| jd k rd| jrd| �t|�� |S r )r0 r2 rD r- �read�lenr3 r4 )r �amountZamount_to_read�datar
r
r
rE s zReadFileChunk.readc C s
d| _ d S )NT�r4 �r r
r
r
r& % s z ReadFileChunk.enable_callbackc C s
d| _ d S �NFrI rJ r
r
r
r ( s z ReadFileChunk.disable_callbackc C s< | j �| j| � | jd k r2| jr2| �|| j � || _d S r )r- r1 r. r3 r4 r2 )r �wherer
r
r
r1 + s zReadFileChunk.seekc C s | j �� d S r )r- �closerJ r
r
r
rM 2 s zReadFileChunk.closec C s | j S r )r2 rJ r
r
r
�tell5 s zReadFileChunk.tellc C s | j S r )r0 rJ r
r
r
�__len__8 s zReadFileChunk.__len__c C s | S r r
rJ r
r
r
� __enter__@ s zReadFileChunk.__enter__c O s | � � d S r )rM )r �argsr$ r
r
r
�__exit__C s zReadFileChunk.__exit__c C s t g �S r )�iterrJ r
r
r
�__iter__F s zReadFileChunk.__iter__)NT)NT)N)r r r r8 � classmethodrC r/ rE r& r r1 rM rN rO rP rR rT r
r
r
r
r) � s$ �
2 � '
r) c @ s" e Zd ZdZddd�Zdd� ZdS )�StreamReaderProgressz<Wrapper for a read only stream that adds progress callbacks.Nc C s || _ || _d S r )�_streamr3 )r �streamr7 r
r
r
r8 R s z StreamReaderProgress.__init__c O s* | j j||�}| jd k r&| �t|�� |S r )rW rE r3 rF )r rQ r$ �valuer
r
r
rE V s
zStreamReaderProgress.read)N)r r r �__doc__r8 rE r
r
r
r
rV O s
rV c @ s4 e Zd Zdd� Zdd� Zdd� Zdd� Zd d
� Zd S ) �OSUtilsc C s t j�|�S r )r; �path�getsize�r r@ r
r
r
�
get_file_size^ s zOSUtils.get_file_sizec C s t j||||dd�S )NF)r& )r) rC )r r@ r+ �sizer7 r
r
r
�open_file_chunk_readera s �z OSUtils.open_file_chunk_readerc C s
t ||�S r )r: )r r@ �moder
r
r
r: f s z OSUtils.openc C s( zt �|� W n tk
r" Y nX dS )z+Remove a file, noop if file does not exist.N)r; �remove�OSErrorr^ r
r
r
� remove_filei s zOSUtils.remove_filec C s t j�||� d S r )�
s3transfer�compat� rename_file)r Zcurrent_filenameZ new_filenamer
r
r
rh r s zOSUtils.rename_fileN)r r r r_ ra r: re rh r
r
r
r
r[ ] s
r[ c @ sH e Zd ZddddgZejjfdd�Zdd� Zd d
� Z d d � Z
d
d� Z dS )�MultipartUploader�SSECustomerKey�SSECustomerAlgorithm�SSECustomerKeyMD5� RequestPayerc C s || _ || _|| _|| _d S r )�_client�_config�_os�
_executor_cls�r �client�config�osutilZ executor_clsr
r
r
r8 � s zMultipartUploader.__init__c C s, i }|� � D ]\}}|| jkr |||< q |S r )�items�UPLOAD_PART_ARGS)r �
extra_argsZupload_parts_args�keyrY r
r
r
�_extra_upload_part_args� s
z)MultipartUploader._extra_upload_part_argsc
C s� | j jf ||d�|��}|d }z| �||||||�}W n^ tk
r� } z@tjddd� | j j|||d� td�|d� ||g�| ���W 5 d } ~ X Y nX | j j
|||d |id
� d S ) N��Bucket�Key�UploadIdzBException raised while uploading parts, aborting multipart upload.T��exc_info)r| r} r~ z Failed to upload {} to {}: {}�/ZParts)r| r} r~ ZMultipartUpload) rn Zcreate_multipart_upload�
_upload_parts� Exception�logger�debugZabort_multipart_uploadr �formatr Zcomplete_multipart_upload)
r r@ �bucketry r7 rx �response� upload_id�parts�er
r
r
� upload_file� sN �� �� � ���z MultipartUploader.upload_filec C s� | � |�}g }| jj} tt�| j�|�t| � ��}
| jj } | j
| d��F} t � | j
||||| ||�}
| �|
td|
d ��D ]}|�|� qzW 5 Q R X |S )N�� max_workers� )rz ro �multipart_chunksize�int�math�ceilrp r_ �float�max_concurrencyrq � functools�partial�_upload_one_part�mapr �append)r r� r@ r� ry r7 rx Zupload_parts_extra_argsr� � part_size� num_partsr� �executorZupload_partial�partr
r
r
r� � s*
��
z MultipartUploader._upload_partsc
C sf | j j} | |||d ||��>}
| jjf |||||
d�|��} | d } | |d�W 5 Q R � S Q R X d S )Nr� )r| r} r~ �
PartNumber�Body�ETag)r� r� )rp ra rn Z upload_part)
r r@ r� ry r� r� rx r7 Z part_number�open_chunk_readerr! r� �etagr
r
r
r� � s&
���z"MultipartUploader._upload_one_partN) r r r rw �
concurrent�futures�ThreadPoolExecutorr8 rz r� r� r� r
r
r
r
ri v s � �
ri c @ s( e Zd ZdZdd� Zdd� Zdd� ZdS ) �
ShutdownQueueaY A queue implementation that can be shutdown.
Shutting down a queue means that this class adds a
trigger_shutdown method that will trigger all subsequent
calls to put() to fail with a ``QueueShutdownError``.
It purposefully deviates from queue.Queue, and is *not* meant
to be a drop in replacement for ``queue.Queue``.
c C s d| _ t�� | _tj�| |�S rK )� _shutdown� threading�Lock�_shutdown_lock�queue�Queue�_init)r �maxsizer
r
r
r� � s
zShutdownQueue._initc C s& | j � d| _t�d� W 5 Q R X d S )NTz The IO queue is now shutdown.)r� r� r� r� rJ r
r
r
�trigger_shutdown� s z ShutdownQueue.trigger_shutdownc C s. | j � | jrtd��W 5 Q R X tj�| |�S )Nz6Cannot put item to queue when queue has been shutdown.)r� r� r( r� r� �put)r �itemr
r
r
r� � s �zShutdownQueue.putN)r r r rZ r� r� r� r
r
r
r
r� � s r� c @ sN e Zd Zejjfdd�Zddd�Zdd� Zdd � Z d
d � Z
d d
� Z dd� Z dS )�MultipartDownloaderc C s* || _ || _|| _|| _t| jj�| _d S r )rn ro rp rq r� � max_io_queue�_ioqueuerr r
r
r
r8
s
z MultipartDownloader.__init__Nc
C sv | j dd��`}t�| j|||||�}|�|�} t�| j|�}
|�|
�} tjj| | gtjj d�} | �
| � W 5 Q R X d S )N� r� )Z return_when) rq r� r� �_download_file_as_futureZsubmit�_perform_io_writesr� r� �waitZFIRST_EXCEPTION�_process_future_results)
r r� ry r@ � object_sizerx r7 �
controllerZdownload_parts_handlerZ parts_futureZio_writes_handlerZ io_future�resultsr
r
r
�
download_file s( �
�
�z!MultipartDownloader.download_filec C s |\}}|D ] }|� � q d S r )�result)r r� �finished�
unfinished�futurer
r
r
r� 1 s z+MultipartDownloader._process_future_resultsc C s� | j j}tt�|t|� ��}| j j}t�| j ||||||�} z0| j
|d��}
t|
�| t|��� W 5 Q R X W 5 | j
� t � X d S )Nr� )ro r� r� r� r� r� r� r� r� �_download_ranger� r� �SHUTDOWN_SENTINELrq �listr� r ) r r� ry r@ r� r7 r� r� r� Zdownload_partialr� r
r
r
r� 6 s � "z,MultipartDownloader._download_file_as_futurec C s: || }||d krd}n || d }d|� d|� �}|S )Nr� r zbytes=�-r
)r r� �
part_indexr� Z start_rangeZ end_range� range_paramr
r
r
�_calculate_range_paramK s z*MultipartDownloader._calculate_range_paramc
s � z�| �|||�}| jj} d }
t| �D ]�} zxt �d� | jj|||d�} t| d |��d� || }
t � �fdd�d�D ] }| j
� |
|f� |
t |�7 }
qvW W �`d S t
jttttfk
r� } z$t jd || | d
d � |}
W Y �q&W 5 d }~X Y q&X q&t|
��W 5 t �d|� X d S ) Nz$EXITING _download_range for part: %szMaking get_object call.)r| r} �Ranger� i @ c s
�� � �S r �rE r
�� buffer_size�streaming_bodyr
r
�<lambda>j � z5MultipartDownloader._download_range.<locals>.<lambda>r� �CRetrying exception caught (%s), retrying request, (attempt %s / %s)Tr )r� r� r� ro �num_download_attemptsr rn �
get_objectrV rS r� r� rF �socket�timeoutrd r r r r )r r� ry r@ r� r� r7 r� r� � max_attempts�last_exception�ir� Z
current_index�chunkr� r
r� r
r� T s^ �
� � � �� z#MultipartDownloader._download_rangec C s� | j �|d���}| j�� }|tkr:t�d� W 5 Q R � d S z |\}}|�|� |�|� W q t k
r� } z tjd|dd� | j�
� � W 5 d }~X Y qX qW 5 Q R X d S )N�wbzCShutdown sentinel received in IO handler, shutting down IO handler.z!Caught exception in IO thread: %sTr ) rp r: r� �getr� r� r� r1 �writer� r� )r r@ rA �task�offsetrH r� r
r
r
r� � s&
�
�
z&MultipartDownloader._perform_io_writes)N)
r r r r� r� r� r8 r� r� r� r� r� r� r
r
r
r
r� s �
�
0r� c @ s( e Zd Zde dde ddfdd�ZdS )�TransferConfigr �
� �d c C s" || _ || _|| _|| _|| _d S r )�multipart_thresholdr� r� r� r� )r r� r� r� r� r� r
r
r
r8 � s
zTransferConfig.__init__N)r r r �MBr8 r
r
r
r
r� � s �r� c @ s� e Zd ZdddddgZdddd d
d d d
dddddddddddddgZd0dd�Zdd� Zd1d d �Zd d � Zd2d d!�Z d"d#� Z
d$d%� Z d&d'� Z d(d)� Z
d*d+� Zd,d-� Zd.d/� ZdS )3�
S3TransferZ VersionIdrk rj rl rm ZACL� CacheControlZContentDispositionZContentEncodingZContentLanguageZ ContentTypeZExpiresZGrantFullControlZ GrantReadZ GrantReadACPZ
GrantWriteACLZMetadataZServerSideEncryptionZ StorageClassZ SSEKMSKeyIdZSSEKMSEncryptionContextZTaggingNc C sF || _ | j jj�d| j� |d kr(t� }|| _|d kr<t� }|| _d S )Nzbefore-call.s3.*) rn �meta�events�register�_update_checksum_contextr� ro r[ �_osutil)r rs rt ru r
r
r
r8 � s
�zS3Transfer.__init__c K s0 |� di �}|� di �}d|kr,d|d d< d S )N�contextZchecksumZrequest_algorithm�header�in)r� )r �paramsr$ �request_contextZchecksum_contextr
r
r
r� � s z#S3Transfer._update_checksum_contextc C s� |dkr i }| � || j� | jjj}|jdtdd� |jdtdd� | j �
|�| j j krl| �
|||||� n| �|||||� dS )z�Upload a file to an S3 object.
Variants have also been injected into S3 client, Bucket and Object.
You don't have to use S3Transfer.upload_file() directly.
Nzrequest-created.s3zs3upload-callback-disable)� unique_idzs3upload-callback-enable)�_validate_all_known_args�ALLOWED_UPLOAD_ARGSrn r� r� Zregister_firstr% Z
register_lastr' r� r_ ro r� �_multipart_upload� _put_object)r r@ r� ry r7 rx r� r
r
r
r� � s(
��
��zS3Transfer.upload_filec C sJ | j j}||d| j �|�|d�� }| jjf |||d�|�� W 5 Q R X d S )Nr )r7 )r| r} r� )r� ra r_ rn Z
put_object)r r@ r� ry r7 rx r� r! r
r
r
r� � s
� ��zS3Transfer._put_objectc C s� |dkr i }| � || j� | �|||�}|tj t� }z| �||||||� W n2 tk
r� tj d|dd� | j
� |� � Y nX | j
� ||� dS )z�Download an S3 object to a file.
Variants have also been injected into S3 client, Bucket and Object.
You don't have to use S3Transfer.download_file() directly.
Nz<Exception caught in download_file, removing partial file: %sTr )
r� �ALLOWED_DOWNLOAD_ARGS� _object_sizer; �extsepr �_download_filer� r� r� r� re rh )r r� ry r@ rx r7 r� Z
temp_filenamer
r
r
r�
s. �� zS3Transfer.download_filec C s8 || j jkr"| �||||||� n| �|||||� d S r )ro r� �_ranged_download� _get_object)r r� ry r@ r� rx r7 r
r
r
r� - s �zS3Transfer._download_filec C s0 |D ]&}||krt d|� dd�|�� ���qd S )NzInvalid extra_args key 'z', must be one of: z, )�
ValueErrorr )r �actual�allowed�kwargr
r
r
r� 7 s
�z#S3Transfer._validate_all_known_argsc C s* t | j| j| j�}|�||||||� d S r )r� rn ro r� r� )r r� ry r@ r� rx r7 Z
downloaderr
r
r
r� ? s � �zS3Transfer._ranged_downloadc
C s� | j j}d }t|�D ]l}z| �|||||�W S tjtttt fk
r~ } z$t
j d| ||dd� | }W Y �qW 5 d } ~ X Y qX qt |��d S )Nr� Tr )
ro r� r �_do_get_objectr� r� rd r r r r� r� r )
r r� ry r@ rx r7 r� r� r� r� r
r
r
r� I s8 �
� � zS3Transfer._get_objectc sf | j jf ||d�|��}t|d |�� | j�|d��(}t� fdd�d�D ]}|�|� qHW 5 Q R X d S )Nr{ r� r� c s
� � d�S )Ni r� r
�r� r
r
r� m r� z+S3Transfer._do_get_object.<locals>.<lambda>r� )rn r� rV r� r: rS r� ) r r� ry r@ rx r7 r� rA r� r
r r
r g s ��zS3Transfer._do_get_objectc C s | j jf ||d�|��d S )Nr{ Z
ContentLength)rn Z head_object)r r� ry rx r
r
r
r� p s �zS3Transfer._object_sizec C s( t | j| j| j�}|�|||||� d S r )ri rn ro r� r� )r r@ r� ry r7 rx Zuploaderr
r
r
r� u s z S3Transfer._multipart_upload)NN)NN)NN)r r r r� r� r8 r� r� r� r� r� r� r� r� r r� r� r
r
r
r
r� � sX � �
�
�
r� )r )1rZ �concurrent.futuresr� r� �loggingr� r; r� r r� r r� Zbotocore.compatr Zbotocore.exceptionsr r Z6botocore.vendored.requests.packages.urllib3.exceptionsr Zs3transfer.compatrf Zs3transfer.exceptionsr r �
__author__� __version__�Handlerr � getLoggerr r� �
addHandlerr� �objectr� r r% r' r� r( r) rV r[ ri r� r� r� r� r� r
r
r
r
�<module>
sH r
q"