�
��ic@s�
dd
lZdd
l
Z
dd
lZdd
lZddlmZ
ddlmZ
ddlm	Z	m
Z

ddlmZ
ddl
mZ
ddlmZmZmZ
d2ZdZdZdZdZd3Zedddg�Zd�Zd
d�
Zedd�Zd�Zd�Z d�Z!d �Z"d!�Z#d"�Z$ej%d#��
Z&d$�Z'd%�Z(d&�Z)d'�Z*d(�Z+d)�Z,ej%d*��
Z-ej%d
d+d,��
Z.d-�Z/d.�Z0d/�Z1d0�Z2d1�Z3d
S(4i����N(
t
namedtuple(
tStopActorExecutionError(tmountingtutils(
tget_env(
tget_target_major_version(tapitCalledProcessErrortrunttmpfstdevtmpfstdevptstsysfstproctcramfstsysvtvfati�i�i�i�iitMountPointstfs_filet
fs_vfstypecCst�d
krt
StS(Nt
8(Rt&_MAGICAL_CONSTANT_MIN_CONTAINER_SIZE_8t&_MAGICAL_CONSTANT_MIN_CONTAINER_SIZE_9(((sV/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/overlaygen.pyt_get_min_container_size`s


c
Cs�t�}
|s#t
jj|�
r'|
Sy3td
d|g�
dj�d}t|�
}Wn4ttfk
r�


t	j
�jdj|
�
�

|
SX|dkr�t	j
�jdj||
��

|
S|
|}|t
kr�t	j
�jdjt
�
�

t
S|S(s�
    Return recommended free space for the target container (+ pkg downloads)

    If the path to the container is set, the returned value is updated to
    reflect already consumed space by the installed container. In case the
    container is bigger than the minimal protected size, return at least
    `_MAGICAL_CONSTANT_MIN_PROTECTED_SIZE`.

    It's not recommended to use this function except official actors managed
    by OAMG group in github.com/oamg/leapp-repository. This function can be
    changed in future, ignoring the deprecation process.

    TODO(pstodulk): this is so far the best trade off between stay safe and do
    do not consume too much space. But need to figure out cost of the time
    consumption.

    TODO(pstodulk): check we are not negatively affected in case of downloaded
    rpms. We want to prevent situations when we say that customer has enough
    space for the first run and after the download of packages we inform them
    they do not have enough free space anymore. Note: such situation can be
    valid in specific cases - e.g. the space is really consumed already e.g. by
    leapp.db that has been executed manytimes.

    :param userspace_path: Path to the userspace container.
    :type userspace_path: str
    :rtype: int
    tdus-sPmxtstdoutisnCannot calculate current container size to estimate correctly required space. Working with the default: {} MiBsCannot calculate the container size - negative size obtained: {}. Estimate the required size based on the default value: {} MiBsnThe size of the container is higher than the expected default. Use the minimal protected size instead: {} MiB.(RtostpathtexistsRtsplittinttOSErrorRRtcurrent_loggertwarningtformatt$_MAGICAL_CONSTANT_MIN_PROTECTED_SIZEtdebug(tuserspace_patht
min_cont_sizet	cont_sizet	prot_size((sV/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/overlaygen.pyt get_recommended_leapp_free_spacefs.	


#













i
cCsTtj
|�
}t|d
�}t|j|j|�
}|
rPt|dd�
S|S(s
    Return the free disk space on given path.

    The default is in bytes, but if convert_to_mibs is True, return MiBs instead.

    Raises OSError if nothing exists on the given `path`.

    :param path: Path to an existing file or directory
    :type path: str
    :param convert_to_mibs: If True, convert the value to MiBs
    :type convert_to_mibs: bool
    :param coefficient: Coefficient to multiply the free space (e.g. 0.9 to have it 10% lower). Max: 1
    :type coefficient: float
    :rtype: int
    i
i(RtstatvfstminRtf_frsizetf_bavail(Rtconvert_to_mibstcoefficienttstattfspace_bytes((sV/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/overlaygen.pyt_get_fspace�s



c	Cs�t|
�
|d
d
kr�dj
d|
d|�}i
dj
d|d|
d|d�d6}td	d�rwd
|d<ntj�j|�

t|d|�

�
ndS(
NisNNot enough space available on {directory}: Needed at least {space_needed} MiB.t	directorytspace_neededsj
The file system hosting the {directory} directory does not contain enough free space to proceed all parts of the in-place upgrade. Note the calculated required free space is the minimum derived from upgrades of minimal systems and the actual needed free space could be higher.
Needed at least: {space_needed} MiB.
Suggested free space: {suggested} MiB (or more).t	suggestedi�tdetailtLEAPP_OVL_SIZEs>The LEAPP_OVL_SIZE environment variable has no effect anymore.tnotetdetails(R2R"RtNoneRR terrorR
(R4R3tmessageR9((sV/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/overlaygen.pyt_ensure_enough_diskimage_space�s	

	


c
Cs�t�}
x�|j
D]�}tjj|j�
r\|jtkr\|
jt	|j|j��

qtjj|j�
r|jd
krt
j�jdj
|j�
�

qqWt|
�
S(NRs8Ignoring vfat {} filesystem mount during upgrade process(tsettfstabRRtisdirRRtOVERLAY_DO_NOT_MOUNTtaddRRR R!R"tlist(tstorage_infotmount_pointstentry((sV/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/overlaygen.pyt_get_mountpoints�s
	

$

$
c

Csd
j|j
dd��
S(Nsroot{}t
/t
_(R"treplace(
t
mountpoint((sV/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/overlaygen.pyt_mount_name�s
cCstj
j|t|
�
�S(
N(RRtjoinRL(t
mounts_dirRK((sV/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/overlaygen.pyt
_mount_dir�s
cCsYxRt|d
t
�

D]>}|ddkr/|ndj|�
}|
j|�
r|SqWdS(Ntreversei����RHs{}/(tsortedtTrueR"t
startswithR:(REtdir_pathtmptmod_mp((sV/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/overlaygen.pyt_get_scratch_mountpoint�s

%

cCsY
tgt
|�
D]}|j^q�
}t||�}tjj|d
�}tdd|g�

t||�
|t	t
|�
}t||�
t|dt
�

|}	i}
x�|D]�}t|dt
dd�
}||kr�|	}n|tkr
d}
tj�j|
||t�
t}nt|||�}tjd|d	t|
|��|
|<q�W|
S(
sz
    Create disk images and loop mount them.

    Ensure to create disk image for each important mountpoint configured
    in fstab (excluding fs types noted in `OVERLAY_DO_NOT_MOUNT`).
    Disk images reflect the free space of related partition/volume. In case
    of partition hosting /var/lib/leapp/* calculate the free space value
    taking `scratch_reserve` into account, as during the run of the tooling,
    we will be consuming the space on the partition and we want to be more
    sure that we do not consume all the space on the partition during the
    execution - so we reduce the risk we affect run of other applications
    due to missing space.

    Note: the partition hosting the scratch dir is expected to be the same
    partition that is hosting the target userspace container, but it does not
    have to be true if the code changes. Right now, let's live with that.

    See `_create_mount_disk_image` docstring for additional more details.

    :param scratch_dir: Path to the scratch directory.
    :type scratch_dir: str
    :param mounts_dir: Path to the directory supposed to be a mountpoint.
    :type mounts_dir: str
    :param storage_info: The StorageInfo message.
    :type storage_info: leapp.models.StorageInfo
    :param scratch_reserve: Number of MB that should be extra reserved in a partition hosting the scratch_dir.
    :type scratch_reserve: Optional[int]
    t
diskimagestrms-rfR.R/gffffff�?s�Image for overlayfs corresponding to the disk mounted at %s would ideally have %d MB, but we truncate it to %d MB to avoid bumping to max file limits.tsourcettarget(RQRGRRWRRRMRt_create_diskimages_dirt_MAGICAL_CONSTANT_OVL_SIZEtlenR=R2RRt_MAX_DISK_IMAGE_SIZE_MBRR tinfot_create_mount_disk_imageRt	LoopMountRO(tscratch_dirRNRDtscratch_reserveRUREt
scratch_mptdisk_images_directoryR4tscratch_disk_sizetresultRKt	disk_sizetmsgtimage((sV/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/overlaygen.pyt_prepare_required_mounts�s,(









	

	
	

ccs�|std
�
�
n|
s#|V
n�t
|
j��
d}|
j|�
}t|�
}|��
tjd|d|d|j��a}tjd|jdt	j
j|j|jd�
���"
t
||
��}|V
WdQXWdQXWdQXWdQXdS(Ns5Root mount point has not been prepared for overlayfs.itnameRZtworkdirR[RH(R
RCtkeystpopRLRtOverlayMountR[t	BindMountRRRMtlstript_build_overlay_mount(t
root_mounttmountstcurrentt
current_mountRmtoverlaytmount((sV/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/overlaygen.pyRtF
s






$

%

cCs'
tj
�jd
�

tjj|
�
r�tj
�jd�

y*tdd|
g�

tj
�jd�

Wq�ttfk
r�
}
tj
�j	d|
t
|�
�
q�Xntdd	�dkr�d	Stj
�jd
|�
t
jdkr�tj|d
tj�


ntj|dtj�


tj
�jd|�
d	S(s$

    Function to cleanup the scratch directory

    If the mounts_dir is a mountpoint, unmount it first.

    :param scratch_dir: Path to the scratch directory.
    :type scratch_dir: str
    :param mounts_dir: Path to the directory supposed to be a mountpoint.
    :type mounts_dir: str
    sCleaning up mountss6Mounts directory is a mounted disk image - Unmounting.s/bin/umounts-flsUnmounted mounted disk image.s!Failed to umount %s - message: %stLEAPP_DEVEL_KEEP_DISK_IMGSt
1Ns*Recursively removing scratch directory %s.iitonexctonerrors)Recursively removed scratch directory %s.(ii(RR R$RRtismountRRRR!tstrRR:tsystversion_infotshutiltrmtreeRt%report_and_ignore_shutil_rmtree_error(RcRNt
e((sV/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/overlaygen.pytcleanup_scratch[
s





&



c
Cs�tj
�jd
|�
dddddd|g}
ytjd|
�

WnZtk
r�
}
tj
�jd	|d
t�

tddj	|�
d
i
t
|�
d6��
n
XdS(s

    Format the specified disk image with Ext4 filesystem.

    The formatted file system is optimized for operations we want to do and
    mainly for the space it needs to take for the initialisation. So use 32MiB
    journal (that's enough for us as we do not plan to do too many operations
    inside) for any size of the disk image. Also the lazy
    initialisation is disabled. The formatting will be slower, but it helps
    us to estimate better the needed amount of the space for other actions
    done later.
    s,Creating ext4 filesystem in disk image at %ss/sbin/mkfs.ext4s-Jssize=32s-Es&lazy_itable_init=0,lazy_journal_init=0s-Ftcmds&Failed to create ext4 filesystem in %stexc_infoR<s#Cannot create Ext4 filesystem in {}R9s
error messageN(RR R$Rtcall_with_oserror_handledRR;RRR
R"R�(tdiskimage_pathR�R�((sV/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/overlaygen.pyt_format_disk_image_ext4�
s








c
Cs�tj
�jd
|�
dddd|g}
ytjd|
�

WnZtk
r�
}
tj
�jd|dt�

td	d
j	|�
di
t
|�
d6��
n
Xd
S(s�
    Format the specified disk image with XFS filesystem.

    Set journal just to 32MiB always as we will not need to do too many operation
    inside, so 32MiB should enough for us.
    s+Creating XFS filesystem in disk image at %ss/sbin/mkfs.xfss-lssize=32ms-fR�s"Failed to create XFS filesystem %sR�R<s"Cannot create XFS filesystem in {}R9s
error messageN(RR R$RR�RR;RRR
R"R�(R�R�R�((sV/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/overlaygen.pyt_format_disk_image_xfs�
s







cCs�|d
kr:tj
�jdjd|
d|��

d
}ntjj|t|
�
�}dddj|�
dd	d
j|�
g}dj|�
}tj
�jd|�
t	j
d
|d|�
tdd�dkr�t|�

n
t
|�

|S(sJ
    Creates the mount disk image and return path to it.

    The disk image is represented by a sparse file which apparent size
    corresponds usually to the free space of a particular partition/volume it
    represents - in this function it's set by `disk_size` parameter, which should
    be int representing the free space in MiBs.

    The created disk image is formatted with XFS (default) or Ext4 FS
    and it's supposed to be used for write directories of an overlayfs built
    above it.

    If the disk_size is lower than 130 MiBs, the disk size is automatically
    set to 130 MiBs to be able to format it correctly.

    The disk image is formatted with Ext4 if (envar) `LEAPP_OVL_IMG_FS_EXT4=1`.

    :param disk_images_directory: Path to the directory where disk images should be stored.
    :type disk_images_directory: str
    :param path: Path to the mountpoint of the original (host/source) partition/volume
    :type path: str
    :param disk_size: Apparent size of the disk img in MiBs
    :type disk_size: int
    :return: Path to the created disk image
    :rtype: str
    i�s�The apparent size for the disk image representing {path} is too small ({disk_size} MiBs) for a formatting. Setting 130 MiBs instead.RRis/bin/ddsif=/dev/zerosof={}sbs=1Mscount=0sseek={}sVPlease ensure that there is enough diskspace on the partition hostingthe {} directory.s%Attempting to create disk image at %sR�thinttLEAPP_OVL_IMG_FS_EXT4t
0R|(RR R!R"RRRMRLR$Rtcall_with_failure_hintRR�R�(RfRRiR�R�R�((sV/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/overlaygen.pyRa�
s"
		


	


cCs�tj
�jd
�

y$tj|
�

tj
�jd�

WnUtk
r�


tj
�jd|
dt�

tdddi
dj	d	|�
d
6��
n
XdS(s.
    Prepares directories for disk images
    sCreating disk images directory.s$Done creating disk images directory.s)Failed to create disk images directory %sR�R<sNFailed to prepare environment for package download while creating directories.R9s9Please ensure that {scratch_dir} is empty and modifiable.RcR�N(
RR R$RtmakedirsRR;RRR
R"(Rctdiskimages_dir((sV/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/overlaygen.pyR\s









cCs�tj
�jd
�

y$tj|
�

tj
�jd�

WnUtk
r�


tj
�jd|
dt�

tdddi
dj	d	|�
d
6��
n
XdS(s)
    Prepares directories for mounts
    sCreating mount directories.s Done creating mount directories.s(Failed to create mounting directories %sR�R<sNFailed to prepare environment for package download while creating directories.R9s9Please ensure that {scratch_dir} is empty and modifiable.RcR�N(
RR R$RR�RR;RRR
R"(RcRN((sV/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/overlaygen.pyt_create_mounts_dirs









c
	cs?tj
d
ddtjj|ddd���}
|
V
WdQXdS(se
    Convenience context manager to ensure bind mounted /var/cache/dnf and removal of the mount.
    RZs/var/cache/dnfR[tvartcachetdnfN(RRrRRRM(toverlay_targettcache_mount((sV/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/overlaygen.pyt_mount_dnf_cache*s	

!
iccsN
tj
�jd
jd|
d|��

z
t|
|�
tdd�dkrbt|
|||�}nt|
|t|�
|�}|j	d�
��}t
jdd	d
dd|j���}|r�t
j
d
|jd|�}	nt
jd|j�
}	|	�8
t||��"}
t|
j�
�

|
V
Wd
QXWd
QXWd
QXWd
QXWd
QXWd
t|
|�
Xd
S(sT	
    Context manager that prepares the source system overlay and yields the mount.

    The in-place upgrade itself requires to do some changes on the system to be
    able to perform the in-place upgrade itself - or even to be able to evaluate
    if the system is possible to upgrade. However, we do not want to (and must not)
    change the original system until we pass beyond the point of not return.

    For that purposes we have to create a layer above the real host file system,
    where we can safely perform all operations without affecting the system
    setup, rpm database, etc. Currently overlay (OVL) technology showed it is
    capable to handle our requirements good enough - with some limitations.

    This function prepares a disk image and an overlay layer for each
    mountpoint configured in /etc/fstab, excluding those with FS type noted
    in the OVERLAY_DO_NOT_MOUNT set. Such prepared OVL images are then composed
    together to reflect the real host filesystem. In the end everything is cleaned.

    The new solution can be now problematic for system with too many partitions
    and loop devices. For such systems we keep for now the possibility of the
    fallback to an old solution, which has however number of issues that are
    fixed by the new design. To fallback to the old solution, set envar:
        LEAPP_OVL_LEGACY=1

    Disk images created for OVL are formatted with XFS by default. In case of
    problems, it's possible to switch to Ext4 FS using:
        LEAPP_OVL_IMG_FS_EXT4=1

    :param mounts_dir: Absolute path to the directory under which all mounts should happen.
    :type mounts_dir: str
    :param scratch_dir: Absolute path to the directory in which all disk and OVL images are stored.
    :type scratch_dir: str
    :param xfs_info: The XFSPresence message.
    :type xfs_info: leapp.models.XFSPresence
    :param storage_info: The StorageInfo message.
    :type storage_info: leapp.models.StorageInfo
    :param mount_target: Directory to which whole source OVL layer should be bind mounted.
                         If None (default), mounting.NullMount is created instead
    :type mount_target: Optional[str]
    :param scratch_reserve: Number of MB that should be extra reserved in a partition hosting the scratch_dir.
    :type scratch_reserve: Optional[int]
    :rtype: mounting.BindMount or mounting.NullMount
    sDCreating source overlay in {scratch_dir} with mounts in {mounts_dir}RcRNtLEAPP_OVL_LEGACYR�R|RHRmtsystem_overlayRZRnR[N(RR R$R"R�RRlt_prepare_required_mounts_oldRGRpRRqR[Rrt	NullMountRtR�R�(RNRctxfs_infoRDtmount_targetRdRvRutroot_overlayR[Ry((sV/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/overlaygen.pytcreate_source_overlay8s"0







$





'cCsotj
|
�
}|j|j|d
d
krkddjd|d|
�}tj�j|�

t|�
�
ndS(NisMNot enough space available for creating required disk images in {directory}. sNeeded: {space_needed} MiBR4R3(	RR*R,R-R"RR R;R
(R4R3R0R<((sV/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/overlaygen.pyt"_ensure_enough_diskimage_space_old�s





cCsVytd
d�}t
|�
}
Wn0tk
rQ


d}
tj�jd||
�
n
X|
S(s?
    Convenient function to retrieve the overlay disk size
    R7t2048isNInvalid "LEAPP_OVL_SIZE" environment variable "%s". Setting default "%d" value(RRt
ValueErrorRR R!(tenv_sizeRi((sV/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/overlaygen.pyt_overlay_disk_size_old�s






cCs�tj
�jd
�

y$tj|
�

tj
�jd�

WnUtk
r�


tj
�jd|
dt�

tdddi
dj	d	|�
d
6��
n
XdS(s.
    Prepares directories for disk images
    sCreating disk images directory.s$Done creating disk images directory.s)Failed to create disk images directory %sR�R<sNFailed to prepare environment for package download while creating directories.R9s9Please ensure that {scratch_dir} is empty and modifiable.RcR�N(
RR R$RR�RR;RRR
R"(RcR�((sV/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/overlaygen.pyt_create_diskimages_dir_old�s









cCs�tj
j|t|
�
�}t�}tj�jd
||�
tj	ddddj
|�
ddj
|�
gdd	j
||��
tj�jd
|�
ytjddd|g�

WnAtk
r�
}
tj�j
d
dt�


tdt|�
�
�
n
X|S(sN
    Creates the mount disk image, for cases when we hit XFS with ftype=0
    s6Attempting to create disk image with size %d MiB at %sR�s/bin/ddsif=/dev/zerosof={}sbs=1Mscount={}R�sMPlease ensure that there is enough diskspace in {} at least {} MiB are neededs,Creating ext4 filesystem in disk image at %ss/sbin/mkfs.ext4s-Fs&Failed to create ext4 filesystem in %sR�R<(RRRMRLR�RR R$RR�R"R�RR;RRR
R�(RfRR�RiR�((sV/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/overlaygen.pyt_create_mount_disk_image_old�s
	
	
'








cs��f
d
�|D�
}|js#|St
�t|j�
}tjj|d�}tdd|g�

t||�
t||�
g|D]}|j	^q�}xat
|jdg
�
D]I}	|	|kr�t||	�}
tj
d|
dt�|	��||	<q�q�W|S(Nc
s1i|]'}
tj
t�|
j��
|
j�qS((RR�ROR(t.0tmount_point(
RN(sV/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/overlaygen.pys
<dictcomp>�s	
RXRYs-rfRHRZR[(tmountpoints_without_ftypeR�R^RRRMRR�R�RR>R�RRbRO(RcRNRER�RhR4RfR�tmount_namesRKRk((
RNsV/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/overlaygen.pyR��s

	








,
(R	R
RRR
RRsvfati(4t
contextlibRR�R�tcollectionsRtleapp.exceptionsR
tleapp.libraries.commonRRtleapp.libraries.common.configRt%leapp.libraries.common.config.versionRtleapp.libraries.stdlibRRRRAR]RRR#R_RRR:R)tFalseR2R=RGRLRORWRltcontextmanagerRtR�R�R�RaR\R�R�R�R�R�R�R�R�(((sV/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/overlaygen.pyt<module>
sP







		@						G	%	 		L		
R