From 0d2c5048b0dcb9b65eca4868587cfe848946eaac Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sun, 21 Jun 2020 12:00:27 +0100 Subject: [PATCH] refactored Alpine image creation --- CONTRIBUTING.md | 10 +- INSTALL | 2 +- LICENSE | 2 +- {debian => archive/debian}/changelog | 0 {debian => archive/debian}/compat | 0 {debian => archive/debian}/control | 0 {debian => archive/debian}/copyright | 0 {debian => archive/debian}/dirs | 0 {debian => archive/debian}/pdf2htmlex.NEWS | 0 {debian => archive/debian}/pdf2htmlex.README | 0 {debian => archive/debian}/pdf2htmlex.TODO | 0 {debian => archive/debian}/rules | 0 {debian => archive/debian}/source/format | 0 buildScripts/Readme.md | 91 +++++++++++++------ buildScripts/buildInstallLocallyAlpine | 8 -- buildScripts/buildInstallLocallyApt | 8 -- buildScripts/createAlpineTarFile | 79 ++++++++++++++++ ...ide => createDockerAlpineImageFromTarFile} | 21 +---- buildScripts/createDockerAlpineImageInside | 37 -------- buildScripts/createDockerUbuntuImageFromDeb | 5 +- buildScripts/createImagesAlpine | 19 ++++ .../{createImages => createImagesApt} | 0 buildScripts/dockerFunctions | 59 ------------ pdf2htmlEX/pdf2htmlEX.1.in | 4 +- pdf2htmlEX/share/base.css.in | 2 +- pdf2htmlEX/share/fancy.css.in | 2 +- pdf2htmlEX/share/manifest | 2 +- pdf2htmlEX/share/pdf2htmlEX.js.in | 2 +- pdf2htmlEX/test/test_local_browser.py | 5 +- 29 files changed, 186 insertions(+), 172 deletions(-) rename {debian => archive/debian}/changelog (100%) rename {debian => archive/debian}/compat (100%) rename {debian => archive/debian}/control (100%) rename {debian => archive/debian}/copyright (100%) rename {debian => archive/debian}/dirs (100%) rename {debian => archive/debian}/pdf2htmlex.NEWS (100%) rename {debian => archive/debian}/pdf2htmlex.README (100%) rename {debian => archive/debian}/pdf2htmlex.TODO (100%) rename {debian => archive/debian}/rules (100%) rename {debian => archive/debian}/source/format (100%) create mode 100755 buildScripts/createAlpineTarFile rename buildScripts/{createDockerAlpineImageOutside => createDockerAlpineImageFromTarFile} (81%) delete mode 100755 buildScripts/createDockerAlpineImageInside create mode 100755 buildScripts/createImagesAlpine rename buildScripts/{createImages => createImagesApt} (100%) delete mode 100644 buildScripts/dockerFunctions diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 72508e2..51159c6 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -23,7 +23,7 @@ A few channels are available to reach the developers, please find the most prope ### The Issue Tracker -The [Issue Tracker](https://github.com/coolwanglu/pdf2htmlEX/issues) +The [Issue Tracker](https://github.com/pdf2htmlEX/pdf2htmlEX/issues) is the best way for [bug reports](#bug-reports), [features requests](#feature-requests) @@ -54,7 +54,7 @@ However do not report issues or submit patches there, since it's terrible to kee pdf2htmlEX is mostly written and maintained by 王璐 (Lu Wang). His email and twitter account can be found in -[README.md](https://github.com/coolwanglu/pdf2htmlEX/blob/master/README.md). +[README.md](https://github.com/pdf2htmlEX/pdf2htmlEX/blob/master/README.md). Please post only messages that do not fit into the above channels, otherwise note that he no longer replies with _Please post your question to the mailing list_ or _Please file an issue at GitHub_, consider your message already replied. @@ -73,7 +73,7 @@ follow the following steps to get it resolved as fast as possible: First of all, did you realize that your question might have been already answered in one of the following places? -- [pdf2htmlEX Wiki](https://github.com/coolwanglu/pdf2htmlEX/wiki) +- [pdf2htmlEX Wiki](https://github.com/pdf2htmlEX/pdf2htmlEX/wiki) - The manpage (run `man pdf2htmlEX`) - Old posts in the [mailing list](#the-mailing-list) or the [issue tracker](#the-issue-tracker) - [Google](http://www.google.com/) @@ -93,7 +93,7 @@ Bugs should always be reported to [the Issue Tracker](#the-issue-tracker). Before you report any bug: - Use the latest git version of pdf2htmlEX, since the issue may have been already fixed. - Search for previous issues (open or closed), to make sure that the issue has not been reported before. -- If pdf2htmlEX crashed, take a look at [this article](https://github.com/coolwanglu/pdf2htmlEX/wiki/Troubleshooting-Crashes). +- If pdf2htmlEX crashed, take a look at [this article](https://github.com/pdf2htmlEX/pdf2htmlEX/wiki/Troubleshooting-Crashes). A good bug report shouldn't leave others needing to chase you up for more information. The developers may be very familiar with the code base of pdf2htmlEX, @@ -110,7 +110,7 @@ If you are not sure, please try to answer the following questions: - What steps will reproduce the issue? — please try to remove unnecessary steps - What's the result and what did you expect? — e.g. you can post screenshots - What error messages did you see? -- What's the affected PDF file and which pages are causing the issue? Create a pull request on [this repo](https://github.com/coolwanglu/pdf2htmlEX-testcase). +- What's the affected PDF file and which pages are causing the issue? Create a pull request on [this repo](https://github.com/pdf2htmlEX/pdf2htmlEX-testcase). Especially for issues regarding building pdf2htmlEX: - Which compiler are you using? diff --git a/INSTALL b/INSTALL index 64e4613..574e0a6 100644 --- a/INSTALL +++ b/INSTALL @@ -1,3 +1,3 @@ For instructions of building the source code, visit: -https://github.com/coolwanglu/pdf2htmlEX/wiki/Building +https://github.com/pdf2htmlEX/pdf2htmlEX/wiki/Building diff --git a/LICENSE b/LICENSE index 90c48ff..1a196de 100644 --- a/LICENSE +++ b/LICENSE @@ -1,4 +1,4 @@ -pdf2htmlEX (https://github.com/coolwanglu/pdf2htmlEX) +pdf2htmlEX (https://github.com/pdf2htmlEX/pdf2htmlEX) Copyright (c) 2012-2014 Lu Wang and other contributors pdf2htmlEX, as a whole package, is licensed under GPLv3 (or any later diff --git a/debian/changelog b/archive/debian/changelog similarity index 100% rename from debian/changelog rename to archive/debian/changelog diff --git a/debian/compat b/archive/debian/compat similarity index 100% rename from debian/compat rename to archive/debian/compat diff --git a/debian/control b/archive/debian/control similarity index 100% rename from debian/control rename to archive/debian/control diff --git a/debian/copyright b/archive/debian/copyright similarity index 100% rename from debian/copyright rename to archive/debian/copyright diff --git a/debian/dirs b/archive/debian/dirs similarity index 100% rename from debian/dirs rename to archive/debian/dirs diff --git a/debian/pdf2htmlex.NEWS b/archive/debian/pdf2htmlex.NEWS similarity index 100% rename from debian/pdf2htmlex.NEWS rename to archive/debian/pdf2htmlex.NEWS diff --git a/debian/pdf2htmlex.README b/archive/debian/pdf2htmlex.README similarity index 100% rename from debian/pdf2htmlex.README rename to archive/debian/pdf2htmlex.README diff --git a/debian/pdf2htmlex.TODO b/archive/debian/pdf2htmlex.TODO similarity index 100% rename from debian/pdf2htmlex.TODO rename to archive/debian/pdf2htmlex.TODO diff --git a/debian/rules b/archive/debian/rules similarity index 100% rename from debian/rules rename to archive/debian/rules diff --git a/debian/source/format b/archive/debian/source/format similarity index 100% rename from debian/source/format rename to archive/debian/source/format diff --git a/buildScripts/Readme.md b/buildScripts/Readme.md index 144c71d..33fae62 100644 --- a/buildScripts/Readme.md +++ b/buildScripts/Readme.md @@ -35,16 +35,6 @@ For most users, you probably really want to simply download one of the [precompiled versions of `pdf2htmlEX`](https://github.com/pdf2htmlEX/pdf2htmlEX/releases): -- [AppImage](https://appimage.org/) : Download, make executable, and - run... - - This will work on most Linuxes, and most recent Windows 10. - -- [Docker](https://www.docker.com/) Image from the [`pdf2htmlEX` Docker - hub](https://hub.docker.com/orgs/pdf2htmlex/repositories). - - This will work on any machine with Docker installed. - - [Debian archive](https://en.wikipedia.org/wiki/Dpkg) : Download, [apt](https://en.wikipedia.org/wiki/APT_(software)) install locally, and run... @@ -55,15 +45,33 @@ For most users, you probably really want to simply download one of the Experienced users of Linux, may be able to repackage the `*.deb` we provide for use with their favourite package management tool. +- [AppImage](https://appimage.org/) : Download, make executable, and + run... + + This will work on most Linuxes, and most recent Windows 10. + + (It will not currently work on MacOS or Alpine based machines). + +- [Docker](https://www.docker.com/) Image from the [`pdf2htmlEX` Docker + hub](https://hub.docker.com/orgs/pdf2htmlex/repositories). + + This will work on any machine with Docker installed. + + (Note: that *advanced* use of `pdf2htmlEX` requires careful attention to + the configuration of various tools, such as fontconfig, iconv and your + locally available fonts use by the poppler and fontforge libraries. The + docker images created by the pdf2htmlEX team might not be as well + configured for *your needs* as a docker created and configured by you) + ### Building yourself -To build `pdf2htmlEX` on your own machine, inside the root directory of -a fresh clone of the +To build `pdf2htmlEX` on a Debian/Apt related machine, inside the root +directory of a fresh clone of the [pdf2htmlEX/pdf2htmlEX](https://github.com/pdf2htmlEX/pdf2htmlEX) repository, type: ``` - ./buildScripts/buildInstallLocally + ./buildScripts/buildInstallLocallyApt ``` This will automatically install all required development tools and @@ -75,6 +83,10 @@ installing `pdf2htmlEX` into /usr/local/bin. [Debian](https://www.debian.org/) based distribution. such as [Ubuntu](https://ubuntu.com/), [Linux Mint](https://linuxmint.com/), etc. +**NOTE:** there is currently an *experimental* build script, +`./buildScripts/buildInstallLocallyAlpine`, for builds in Alpine +environments. + ## The problem To provide its full functionality, the `pdf2htmlEX` sources make direct @@ -126,7 +138,7 @@ Each script can be used individually to re-run a particular step if needed. Typically, most users, will run one of the following "top-level" scripts: -1. **`buildInstallLocally`** +1. **`buildInstallLocallyApt`** (**`buildInstallLocallyAlpine`**) This will automate: @@ -138,10 +150,16 @@ Typically, most users, will run one of the following "top-level" scripts: 3. compile and install `pdf2htmlEX`. -2. **`createImages`** + The `*Apt` script will build on any machine which uses the + `apt`/`apt-get` command. - Following a successful `buildInstallLocally`, this will create the - following images: + The `*Alpine` script will build on any machine which uses the + `apk` command (Alpine). + +2. **`createImagesApt`** (**`createImagesAlpine`**) + + Following a successful `buildInstallLocallyApt`, the `createImagesApt` + shell script will create the following images: 1. AppImage @@ -149,10 +167,18 @@ Typically, most users, will run one of the following "top-level" scripts: 3. Debian archive + Following a successful `buildInstallLocallyAlpine`, the + `createImagesAlpine` shell script will create the following images: + + 1. Alpine tar file + + 2. Docker image + 3. **`runTests`** - Following a successful `buildInstallLocally`, this will run the various - 'local' tests reporting errors as they occur. + Following a successful `buildInstallLocallyApt` (or + `buildInstallLocallyAlpine` ), the `runTests` shell script will run the + various 'local' tests reporting errors as they occur. When run in [Travis-ci](https://travis-ci.org/), failing browser tests will *not* fail the overall Travis build, but will instead upload the @@ -192,6 +218,10 @@ Typically, most users, will run one of the following "top-level" scripts: ensures that any versions of Poppler already installed by the user, are not broken by the user's installation of `pdf2htmlEX`. +- **`createAlpineTarFile`**: Using an already compiled version of `pdf2htmlEX`, + installs it and `popper-data` into a tar file suitable for use in any + Alpine environment. + - **`createAppImage`**: Using an already compiled version of `pdf2htmlEX`, installs it and `popper-data` into an AppImage. @@ -199,15 +229,28 @@ Typically, most users, will run one of the following "top-level" scripts: `pdf2htmlEX`, installs it and `poppler-data` into a Debian archive (`*.deb`). +- **`createDockerAlpineImageFromTarFile`**: Installs the Alpine tar file + archive of `pdf2htmlEX` created by `createAlpineTarFile` into an Alpine + Docker image. + - **`createDockerUbuntuImageFromDeb`**: Installs the Debian archive of `pdf2htmlEX` created by `createDebianPackage` into a Docker image. -- **`getBuildTools`**: Locally `apt` installs all development *tools* +- **`getBuildToolsAlpine`**: Locally `apk` installs all development + *tools* required to build `pdf2htmlEX`. + +- **`getBuildToolsApt`**: Locally `apt` installs all development *tools* required to build `pdf2htmlEX`. -- **`getDevLibraries`**: Locally `apt` installs all development +- **`getDevLibrariesAlpine`**: Locally `apk` installs all development *libraries* required to build `pdf2htmlEX`. +- **`getDevLibrariesApt`**: Locally `apt` installs all development + *libraries* required to build `pdf2htmlEX`. + + This script provides a definitive list of all libraries required to run + `pdf2htmlEX`. + This script provides a definitive list of all libraries required to run `pdf2htmlEX`. @@ -236,7 +279,7 @@ Typically, most users, will run one of the following "top-level" scripts: The `PDF2HTMLEX_PREFIX` variable is specified in the `versionEnvs` script. -- **`rutTests`**: Runs the tests located in the +- **`runTests`**: Runs the tests located in the `pdf2htmlEX/pdf2htmlEX/test` directory. See the `pdf2htmlEX/pdf2htmlEx/test` directory's Readme file for details. @@ -279,10 +322,6 @@ Typically, most users, will run one of the following "top-level" scripts: used by the `uploadGitHubRelease` script as the contents of the release message, as visible to the user, for the 'continuous' release section. -- **`dockerFunctions`**: A collection of shell functions used by the - `uploadDockerImage` script to automate the upload of the Docker images - to Docker Hub. - - **`listFilesByChangeTime`**: A simple shell script which lists the files in the buildScripts directory by most recently changed files first. diff --git a/buildScripts/buildInstallLocallyAlpine b/buildScripts/buildInstallLocallyAlpine index 8fe6d2f..5a535bc 100755 --- a/buildScripts/buildInstallLocallyAlpine +++ b/buildScripts/buildInstallLocallyAlpine @@ -47,11 +47,3 @@ set -ev ./buildScripts/buildPdf2htmlEX ./buildScripts/installPdf2htmlEX - -#./buildScripts/createAppImage - -#./buildScripts/createDockerUbuntuImageFromDeb - -#./buildScripts/uploadGitHubRelease - -#./buildScripts/uploadDockerImage diff --git a/buildScripts/buildInstallLocallyApt b/buildScripts/buildInstallLocallyApt index 2dcded2..a26afc4 100755 --- a/buildScripts/buildInstallLocallyApt +++ b/buildScripts/buildInstallLocallyApt @@ -47,11 +47,3 @@ set -ev ./buildScripts/buildPdf2htmlEX ./buildScripts/installPdf2htmlEX - -#./buildScripts/createAppImage - -#./buildScripts/createDockerUbuntuImageFromDeb - -#./buildScripts/uploadGitHubRelease - -#./buildScripts/uploadDockerImage diff --git a/buildScripts/createAlpineTarFile b/buildScripts/createAlpineTarFile new file mode 100755 index 0000000..5d99e56 --- /dev/null +++ b/buildScripts/createAlpineTarFile @@ -0,0 +1,79 @@ +#!/bin/sh + +# This shell script creates a docker image from an existing pdf2htmlEX +# +# This is the part which can run *inside* a docker container. + +echo "" +echo "-------------------------------------------------------------------" +echo "CREATING pdf2htmlEX Docker Image (run inside a docker container)" +echo " (based on Alpine linux $BUILD_DIST)" +echo "-------------------------------------------------------------------" +echo "" + +# Collect everything that will be needed... + +# source buildScripts/reSourceVersionEnvs +. buildScripts/reSourceVersionEnvs + +export ALPINE_NAME="pdf2htmlEX-$PDF2HTMLEX_NAME" + +echo "export ALPINE_NAME=\"$ALPINE_NAME\"" >> buildScripts/reSourceVersionEnvs + +set -ev + +mkdir -p imageBuild/alpineTarDir + +cd pdf2htmlEX/build + +sudo rm -rf install_manifest.txt + +make install DESTDIR=../../imageBuild/alpineTarDir + +cd ../../poppler-data + +make install \ + prefix=$PDF2HTMLEX_PREFIX \ + datadir=$PDF2HTMLEX_PREFIX/share/pdf2htmlEX \ + DESTDIR=../imageBuild/alpineTarDir + +cd ../imageBuild + +tar czvf $ALPINE_TARFILE_NAME.tar.gz alpineTarDir + +cat < $ALPINE_NAME.install +#!/bin/sh + +# This (alpine) shell script installs a locally existing tar.gz file of +# the pdf2htmlEX binaries compiled on Alpine $DIST. +# +# You MUST have root/sudo privileges to run this file. +# + +# We start by installing all of the (known) required runtime dependencies +# +apk update +# +apk add --no-cache \ + tar \ + libstdc++ \ + libgcc \ + gnu-libiconv \ + gettext \ + glib \ + freetype \ + fontconfig \ + cairo \ + libpng \ + libjpeg-turbo \ + libxml2 + +# Now we install the (Alpine $DIST) compiled pdf2htmlEX binaries and +# configuration files. +# +cd / +# +tar xvf $ALPINE_NAME.tar.gz --strip-comonents=1 + +ALPINE_INSTALL + diff --git a/buildScripts/createDockerAlpineImageOutside b/buildScripts/createDockerAlpineImageFromTarFile similarity index 81% rename from buildScripts/createDockerAlpineImageOutside rename to buildScripts/createDockerAlpineImageFromTarFile index ff6fa44..2ff5954 100755 --- a/buildScripts/createDockerAlpineImageOutside +++ b/buildScripts/createDockerAlpineImageFromTarFile @@ -16,15 +16,14 @@ echo "" # source buildScripts/reSourceVersionEnvs . buildScripts/reSourceVersionEnvs -# source buildScripts/dockerFunctions -. buildScripts/dockerFunctions - set -ev mkdir -p imageBuild/dockerDir cd imageBuild/dockerDir +cp ../$ALPINE_NAME.* . + if [ -z "$DOCKER_FROM" ]; then echo "" read -p "Enter the docker image for the 'from' base: " DOCKER_FROM @@ -54,21 +53,9 @@ echo "export DOCKER_NAME=\"$DOCKER_NAME\"" >> ../../buildScripts/reSourc cat > Dockerfile <> ../../buildScripts/reSourc cat > Dockerfile <" | awk '{print $3}' \ - | xargs -I '{}' echo '{}' >> DEPSFILE - done - - DEPS=$(cat DEPSFILE | sort | uniq) - - for FILE in $DEPS ; do - if [ -e $FILE ] && [[ $(readlink -f $FILE)/ != $PWD/* ]] ; then - cp -v --parents -rfL $FILE ./ || true - fi - done - rm -f DEPSFILE -} - -# Delete blacklisted files. Our blacklisted files are the shared libraries -# extracted from the "base" docker image -# -delete_blacklisted() -{ - docker run --rm -it --entrypoint=find $DOCKER_FROM -iname "*.so.*" \ - | xargs -i basename {} | sed 's/\r//g' | sed 's/\(so\.[0-9]\+\).*$/\1/' \ - | sort | uniq > BLACKLISTED_FILES - - for FILE in $(cat BLACKLISTED_FILES | tr "\r\n" " ") ; do - FILES="$(find . -name "${FILE}*" -not -path "./usr/optional/*")" - for FOUND in $FILES ; do - rm -vf "$FOUND" "$(readlink -f "$FOUND")" - done - done - - rm BLACKLISTED_FILES - - # Do not bundle developer stuff - rm -rf usr/include || true - rm -rf usr/lib/cmake || true - rm -rf usr/lib/pkgconfig || true - find . -name '*.la' | xargs -i rm {} -} - diff --git a/pdf2htmlEX/pdf2htmlEX.1.in b/pdf2htmlEX/pdf2htmlEX.1.in index d3cd881..570997c 100644 --- a/pdf2htmlEX/pdf2htmlEX.1.in +++ b/pdf2htmlEX/pdf2htmlEX.1.in @@ -359,7 +359,7 @@ pdf2htmlEX is written by Lu Wang .SH SEE ALSO .TP Home page -https://github.com/coolwanglu/pdf2htmlEX +https://github.com/pdf2htmlEX/pdf2htmlEX .TP pdf2htmlEX Wiki -https://github.com/coolwanglu/pdf2htmlEX/wiki +https://github.com/pdf2htmlEX/pdf2htmlEX/wiki diff --git a/pdf2htmlEX/share/base.css.in b/pdf2htmlEX/share/base.css.in index 515134d..7b2feac 100644 --- a/pdf2htmlEX/share/base.css.in +++ b/pdf2htmlEX/share/base.css.in @@ -2,7 +2,7 @@ /*! * Base CSS for pdf2htmlEX * Copyright 2012,2013 Lu Wang - * https://github.com/coolwanglu/pdf2htmlEX/blob/master/share/LICENSE + * https://github.com/pdf2htmlEX/pdf2htmlEX/blob/master/share/LICENSE */ /* Part 1: Web Page Layout: Free to modify, except for a few of them which are required by pdf2htmlEX.js, see the comments */ #sidebar { /* Sidebar */ diff --git a/pdf2htmlEX/share/fancy.css.in b/pdf2htmlEX/share/fancy.css.in index bdba3f6..138b6f2 100644 --- a/pdf2htmlEX/share/fancy.css.in +++ b/pdf2htmlEX/share/fancy.css.in @@ -2,7 +2,7 @@ /*! * Fancy styles for pdf2htmlEX * Copyright 2012,2013 Lu Wang - * https://github.com/coolwanglu/pdf2htmlEX/blob/master/share/LICENSE + * https://github.com/pdf2htmlEX/pdf2htmlEX/blob/master/share/LICENSE */ @keyframes fadein { from { opacity:0;} to { opacity:1;} } @-webkit-keyframes fadein { from { opacity:0;} to { opacity:1;} } diff --git a/pdf2htmlEX/share/manifest b/pdf2htmlEX/share/manifest index b03560f..0380867 100644 --- a/pdf2htmlEX/share/manifest +++ b/pdf2htmlEX/share/manifest @@ -17,7 +17,7 @@ # Declaration - Do not modify """ - + diff --git a/pdf2htmlEX/share/pdf2htmlEX.js.in b/pdf2htmlEX/share/pdf2htmlEX.js.in index a678759..9423ce9 100644 --- a/pdf2htmlEX/share/pdf2htmlEX.js.in +++ b/pdf2htmlEX/share/pdf2htmlEX.js.in @@ -2,7 +2,7 @@ /** * @license pdf2htmlEX.js: Core UI functions for pdf2htmlEX * Copyright 2012,2013 Lu Wang and other contributors - * https://github.com/coolwanglu/pdf2htmlEX/blob/master/share/LICENSE + * https://github.com/pdf2htmlEX/pdf2htmlEX/blob/master/share/LICENSE */ /* diff --git a/pdf2htmlEX/test/test_local_browser.py b/pdf2htmlEX/test/test_local_browser.py index 7c67a3d..62a3657 100755 --- a/pdf2htmlEX/test/test_local_browser.py +++ b/pdf2htmlEX/test/test_local_browser.py @@ -43,4 +43,7 @@ class test_local_browser(BrowserTests, unittest.TestCase): self.browser.save_screenshot(png_file) if __name__ == '__main__': - unittest.main() + # turn warnings OFF to prevent 'ResourceWarning: unclosed