From 3f45343ba4e40f266457c7d3dedc62931b7988ed Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Fri, 5 Jun 2020 11:44:33 +0100 Subject: [PATCH 01/69] added more comprehensive documentation for the build scripts --- buildScripts/Readme.md | 253 ++++++++++++++++++ buildScripts/cleanPdf2htmlEX | 6 - buildScripts/getPdf2htmlEX | 25 -- .../runLocalTests => buildScripts/runTests | 0 4 files changed, 253 insertions(+), 31 deletions(-) create mode 100644 buildScripts/Readme.md delete mode 100755 buildScripts/cleanPdf2htmlEX delete mode 100755 buildScripts/getPdf2htmlEX rename testScripts/runLocalTests => buildScripts/runTests (100%) diff --git a/buildScripts/Readme.md b/buildScripts/Readme.md new file mode 100644 index 0000000..b794c52 --- /dev/null +++ b/buildScripts/Readme.md @@ -0,0 +1,253 @@ +# Building pdf2htmlEX + +Because of its intimate use of *specific* versions of both Poppler and +FontForge, cleanly building `pdf2htmlEX` is rather more complex than +normal. + +The (bash) scripts in this directory help automate this mutli-stage +process. + +For all but the most experienced programmers, we *strongly* encourage you +to use these scripts to build `pdf2htmlEX`. + +--- + +**Table of contents** + +- [TL;DR ...](#tldr-) + - [Downloading precompiled versions](#http://localhost:5000/Readme.md#downloading-precompiled-versions-downloads) + - [Building yourself](#http://localhost:5000/Readme.md#building-yourself) +- [The gory details ...](#http://localhost:5000/Readme.md#the-gory-details-) + - [Top-level scripts](#http://localhost:5000/Readme.md#top-level-scripts) + - [Individual steps](#http://localhost:5000/Readme.md#individual-steps) + - [Helper files and scripts](http://localhost:5000/Readme.md#helper-files-and-scripts) +- [Yet more details?](http://localhost:5000/Readme.md#yet-more-details) + +--- + +## TL;DR ... + +### Downloading precompiled versions {#Downloads} + +For most users, you probably really want to simply download one of the +[precompiled versions of +`pdf2htmlEX`](https://github.com/pdf2htmlEX/pdf2htmlEX/releases): + +- [AppImage](https://appimage.org/) : Download, make executable, and + run... + + This will work on most Linuxes, and most recent Windows 10. + +- [Docker](https://www.docker.com/) Image from the [`pdf2htmlEX` Docker + hub](https://hub.docker.com/orgs/pdf2htmlex/repositories). + + This will work on any machine with Docker installed. + +- [Debian archive](https://en.wikipedia.org/wiki/Dpkg) : Download, + [apt](https://en.wikipedia.org/wiki/APT_(software)) install locally, + and run... + + This will work on any [Debian](https://www.debian.org/) based and most + recent Windows 10 machines. + + Experienced users of Linux, may be able to repackage this `*.deb` for + use with their favourite package management tool. + +### Building yourself + +To build `pdf2htmlEX` on your own machine, inside the root directory of +a fresh clone of the +[pdf2htmlEX/pdf2htmlEX](https://github.com/pdf2htmlEX/pdf2htmlEX) +repository, type: + +``` + ./buildScripts/buildInstallLocally +``` + +This will automatically install all required development tools and +libraries, and then proceed to download and statically compile the +required versions of both Poppler and FontForge before compiling and +installing `pdf2htmlEX` into /usr/local/bin. + +**NOTE:** at the moment this will **only** work on machines with a +[Debian](https://www.debian.org/) based distribution. such as +[Ubuntu](https://ubuntu.com/), [Linux Mint](https://linuxmint.com/), etc. + +## The gory details ... + +The bash scripts in this directory automate the download, build, install, +test and upload steps required to provide a complete build/test/release +cycle of `pdf2htmlEX`. + +Each script can be used individually to re-run a particular step if needed. + +### Top-level scripts + +However, typically, most users, will run one of the following +"top-level" scripts: + +1. **`buildInstallLocally`** + + This will automate: + + 1. the installation of all required development tools + and libraries, + + 2. download and statically compile the required versions of both + Poppler and FontForge, + + 3. compile and install `pdf2htmlEX`. + +2. **`createImages`** + + Following a successful `buildInstallLocally`, this will create the + following images: + + 1. AppImage + + 2. Docker image + + 3. Debian archive + +3. **`runTests`** + + Following a successful `buildInstallLocally`, this will run the various + 'local' tests reporting errors as they occur. + + When run in [Travis-ci](https://travis-ci.org/), failing browser tests + will *not* fail the overall Travis build, but will instead upload the + test results to the GitHub Release page for later review. + +4. **`uploadImages`** + + Following successful `buildInstallLocally`, `createImages` and + `runTests`, this will automate the upload of the various artefacts to + the `pdf2htmlEX` releases page, LaunchPad ppa and docker hub + repository. + + **Note** that this step requires the user to enter passwords for each + of the respective services. *Most* users will not need (or be able) to + run this step. + +5. **`travisLinuxDoItAll`** + + This script is used by the `.travis.yml` configuration to build, test + and upload a complete `pdf2htmlEX` release cycle. It is essentially a + compendium of all of the build scripts in the correct order. + +### Individual steps + +- **`buildFontforge`**: Compiles a *static* version of `libfontforge` for + use by `pdf2htmlEX`. + +- **`buildPdf2htmlEX`**: Compiles and links `pdf2htmlEX`. + +- **`buildPoppler`**: Compiles a *static* version of `libpoppler` and + `libpopper-glib` for use by `pdf2htmlEX`. + +- **`createAppImage`**: Using an already compiled version of `pdf2htmlEX`, + installs it and `popper-data` into an AppImage. + +- **`createDebianPackage`**: Using an already compiled version of + `pdf2htmlEX`, installs it and `poppler-data` into a Debian archive + (`*.deb`). + +- **`createDockerImage`**: Using an already compile version of + `pdf2htmlEX`, installs it and `poppler-data` into a Docker image. + +- **`getBuildTools`**: Locally `apt` installs all development *tools* + required to build `pdf2htmlEX`. + +- **`getDevLibraries`**: Locally `apt` installs all development + *libraries* required to build `pdf2htmlEX`. + + This script provides a definitive list of all libraries required to run + `pdf2htmlEX`. + +- **`getFontforge`**: Downloads and unpacks the version of FontForge specified in the + `FONTFORGE_VERSION` environment variable into the + `pdf2htmlEX/fontoforge` directory. + + The `FONTFORGE_VERSION` variable is specified in the `versionEnvs` + script. + +- **`getPoppler`**: Downloads and unpacks the version of Poppler specified in the + `POPPLER_VERSION` environment variable into the `pdf2htmlEX/poppler` + directory. + + The `POPPLER_VERSION` variable is specified in the `versionEnvs` script. + + The `getPoppler` script also downloads and unpacks the most recent + version of `poppler-data`. Since `poppler-data` does not change very + often, the correct version of `poppler-data` is specified in the + `getPoppler` script itself. + +- **`installPdf2htmlEX`**: Installs an already compiled version of + `pdf2htmlEX` and `poppler-data` into the location specified by the + `PDF2HTMLEX_PREFIX` environment variable. + + The `PDF2HTMLEX_PREFIX` variable is specified in the `versionEnvs` + script. + +- **`rutTests`**: Runs the tests located in the + `pdf2htmlEX/pdf2htmlEX/test` directory. See the + `pdf2htmlEX/pdf2htmlEx/test` directory's Readme file for details. + +- **`uploadDebianArchive`**: Upload the `pdf2htmlEX` Debian archive to + both the GitHub releases page, as well as the LaunchPad ppa. + + This script has not yet been implemented. + +- **`uploadDockerImage`**: Upload the `pdf2htmlEX` Docker image to + Docker hub repository associated to the docker hub users specified in + the `DOCKER_USERNAME` environement variable. + + Unless the `DOCKER_USERNAME` and `DOCKER_PASSWORD` environment variables + are pre-defined, this script will prompt the user for the respective + values. + +- **`uploadGitHubRelease`**: Upload the `pdf2htmlEX` artefacts (AppImage, + Debian archive, test results, etc) to the *continuous* section of the + release page associated with the `TRAVIS_REPO_SLUG` (user/project) + environment variable. + + Unless the `GITHUB_USERNAME`, `GITHUB_TOKEN`, and `TRAVIS_REPO_SLUG` + (user/project) environment variables are pre-defined, this script will + prompt the user for the respective values. + +### Helper files and scripts + +- **`versionEnvs`**: Specifies all of the evnironment variables required + for a standard build of `pdf2htmlEX`. Changes in this script effect + *all* of the other build scripts. + +- **`reSourceVersionEnvs`**: This bash script is automatically generated + by the build scripts as they are run. It records the values of all + important environment variables required by the buildScripts. It is + typcically `source`d by each script before it preforms any actions. + +- **`reportEnvs`**: Echos all important enviroment variables to the + console. This script is used by the top-level scripts to ensure the + current environment variables are listed before each build. + +- **`uploadGitHubReleaseDSL`**: A collection of bash functions used by the + `uploadGitHubRelease` script to automate the upload of release artefacts. + +- **`uploadGitHubReleaseMessage`**: The contents of this *text* file is + used by the `uploadGitHubRelease` script as the contents of the release + message, as visible to the user, for the 'continuous' release section. + +- **`dockerFunctions`**: A collection of bash functions used by the + `uploadDockerImage` script to automate the upload of the Docker images + to Docker Hub. + +- **`listFilesByChangeTime`**: A simple bash script which lists the files + in the buildScripts directory by most recently changed files first. + +- **`Readme.md`**: This read me file. + +## Yet more details? + +The various bash script files are meant to be fairly readable. They +contain additional comments about what each step is meant to be doing. + diff --git a/buildScripts/cleanPdf2htmlEX b/buildScripts/cleanPdf2htmlEX deleted file mode 100755 index a01e378..0000000 --- a/buildScripts/cleanPdf2htmlEX +++ /dev/null @@ -1,6 +0,0 @@ -#!/bin/bash - -# This bash script clean all pdf2htmlEX build products - -cd pdf2htmlEX -rm -rf build pdf2htmlEX.1 share/*.css share/*.js share/*.min.* src/pdf2htmlEX-config.h src/util/css_const.h diff --git a/buildScripts/getPdf2htmlEX b/buildScripts/getPdf2htmlEX deleted file mode 100755 index 3084d1a..0000000 --- a/buildScripts/getPdf2htmlEX +++ /dev/null @@ -1,25 +0,0 @@ -#!/bin/bash - -# This bash script automates the process of getting the original pdf2htmlEX -# source - -PDF2HTMLEX_BRANCH=update-poppler - -echo "" -echo "-------------------------------------------------------------------" -echo "GETTING pdf2htmlEX sources (using wget)" -echo " (PDF2HTMLEX_BRANCH: [$PDF2HTMLEX_BRANCH])" -echo "-------------------------------------------------------------------" -echo "" - -set -ev - -wget https://codeload.github.com/stephengaito/pdf2htmlEX/zip/$PDF2HTMLEX_BRANCH - -mv $PDF2HTMLEX_BRANCH $PDF2HTMLEX_BRANCH.zip - -sudo apt install unzip - -unzip $PDF2HTMLEX_BRANCH.zip - -mv pdf2htmlEX-$PDF2HTMLEX_BRANCH pdf2htmlEX diff --git a/testScripts/runLocalTests b/buildScripts/runTests similarity index 100% rename from testScripts/runLocalTests rename to buildScripts/runTests From d49260afe7b385a9a17cd8a8057fcf8be99be877 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Fri, 5 Jun 2020 11:47:46 +0100 Subject: [PATCH 02/69] corrected table of contents --- buildScripts/Readme.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/buildScripts/Readme.md b/buildScripts/Readme.md index b794c52..c558f24 100644 --- a/buildScripts/Readme.md +++ b/buildScripts/Readme.md @@ -15,13 +15,13 @@ to use these scripts to build `pdf2htmlEX`. **Table of contents** - [TL;DR ...](#tldr-) - - [Downloading precompiled versions](#http://localhost:5000/Readme.md#downloading-precompiled-versions-downloads) - - [Building yourself](#http://localhost:5000/Readme.md#building-yourself) -- [The gory details ...](#http://localhost:5000/Readme.md#the-gory-details-) - - [Top-level scripts](#http://localhost:5000/Readme.md#top-level-scripts) - - [Individual steps](#http://localhost:5000/Readme.md#individual-steps) - - [Helper files and scripts](http://localhost:5000/Readme.md#helper-files-and-scripts) -- [Yet more details?](http://localhost:5000/Readme.md#yet-more-details) + - [Downloading precompiled versions](#downloading-precompiled-versions-downloads) + - [Building yourself](#building-yourself) +- [The gory details ...](#the-gory-details-) + - [Top-level scripts](#top-level-scripts) + - [Individual steps](#individual-steps) + - [Helper files and scripts](#helper-files-and-scripts) +- [Yet more details?](#yet-more-details) --- From d4c172d8ce199cfc69b81d57800be409a735d870 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Fri, 5 Jun 2020 11:55:54 +0100 Subject: [PATCH 03/69] minor corrections --- buildScripts/Readme.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/buildScripts/Readme.md b/buildScripts/Readme.md index c558f24..bca2a5d 100644 --- a/buildScripts/Readme.md +++ b/buildScripts/Readme.md @@ -83,8 +83,7 @@ Each script can be used individually to re-run a particular step if needed. ### Top-level scripts -However, typically, most users, will run one of the following -"top-level" scripts: +Typically, most users, will run one of the following "top-level" scripts: 1. **`buildInstallLocally`** From 49712c9ec2fcee0feadcde68d31c24f7a876d418 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Fri, 5 Jun 2020 12:06:51 +0100 Subject: [PATCH 04/69] minor corrections --- buildScripts/Readme.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/buildScripts/Readme.md b/buildScripts/Readme.md index bca2a5d..091bc4c 100644 --- a/buildScripts/Readme.md +++ b/buildScripts/Readme.md @@ -137,13 +137,21 @@ Typically, most users, will run one of the following "top-level" scripts: ### Individual steps - **`buildFontforge`**: Compiles a *static* version of `libfontforge` for - use by `pdf2htmlEX`. + use by `pdf2htmlEX`. + + Statically linking `libfontforge` into `phd2htmlEX` ensures that any + versions of FontForge already installed by the user, are not broken by + the user's installation of `pdf2htmlEX`. - **`buildPdf2htmlEX`**: Compiles and links `pdf2htmlEX`. - **`buildPoppler`**: Compiles a *static* version of `libpoppler` and `libpopper-glib` for use by `pdf2htmlEX`. + Statically linking `libpoppler` and `libpoppler-glib` into `phd2htmlEX` + ensures that any versions of Poppler already installed by the user, are + not broken by the user's installation of `pdf2htmlEX`. + - **`createAppImage`**: Using an already compiled version of `pdf2htmlEX`, installs it and `popper-data` into an AppImage. From a5209ace824481ba31711d55563e31ac6f1fb80a Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sat, 6 Jun 2020 06:18:14 +0100 Subject: [PATCH 05/69] turned on simple tests and upload from travis --- buildScripts/runTests | 4 ++-- buildScripts/travisLinuxDoItAll | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/buildScripts/runTests b/buildScripts/runTests index 49ff25d..4a1f355 100755 --- a/buildScripts/runTests +++ b/buildScripts/runTests @@ -8,9 +8,9 @@ cd pdf2htmlEX/test # The following is only needed for the local browser tests # -./installAutomaticTestSoftware +#./installAutomaticTestSoftware ./runLocalTests -./runLocalBrowserTests +#./runLocalBrowserTests diff --git a/buildScripts/travisLinuxDoItAll b/buildScripts/travisLinuxDoItAll index 13577c9..681e7ab 100755 --- a/buildScripts/travisLinuxDoItAll +++ b/buildScripts/travisLinuxDoItAll @@ -28,9 +28,9 @@ export DEBIAN_FRONTEND=noninteractive ./buildScripts/buildFontforge ./buildScripts/buildPdf2htmlEX ./buildScripts/installPdf2htmlEX -#./testScripts/runLocalTests +./buildScripts/runTests ./buildScripts/createAppImage ./buildScripts/createDebianPackage ./buildScripts/createDockerImage -#./buildScripts/uploadGitHubRelease -#./buildScripts/uploadDockerImage +./buildScripts/uploadGitHubRelease +./buildScripts/uploadDockerImage From 0d5d7ad721e1522b539fbb524cdc0810ccf10932 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sat, 6 Jun 2020 06:33:45 +0100 Subject: [PATCH 06/69] corrected travis configuration --- .travis.yml | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/.travis.yml b/.travis.yml index b2a16e5..47709a7 100644 --- a/.travis.yml +++ b/.travis.yml @@ -19,17 +19,14 @@ jobs: include: - stage: build-linux os: linux -# env: -# global: DOCKER_FROM="ubuntu:18.04" -# services: -# - docker dist: bionic + env: + global: DOCKER_FROM="ubuntu:18.04" + services: + - docker script: - ./buildScripts/travisLinuxDoItAll -# - stage: test -# os: -# - linux -# script: ./testScripts/runLocalTests + #deploy: # provider: releases # api_key: From 2faa2c022aa0b0074663da09571115bb3394adb4 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sat, 6 Jun 2020 07:03:26 +0100 Subject: [PATCH 07/69] corrected travis build --- .travis.yml | 3 +-- buildScripts/travisLinuxDoItAll | 2 +- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/.travis.yml b/.travis.yml index 47709a7..c8aaee5 100644 --- a/.travis.yml +++ b/.travis.yml @@ -20,8 +20,7 @@ jobs: - stage: build-linux os: linux dist: bionic - env: - global: DOCKER_FROM="ubuntu:18.04" + env: DOCKER_FROM="ubuntu:18.04" services: - docker script: diff --git a/buildScripts/travisLinuxDoItAll b/buildScripts/travisLinuxDoItAll index 681e7ab..22771b2 100755 --- a/buildScripts/travisLinuxDoItAll +++ b/buildScripts/travisLinuxDoItAll @@ -7,7 +7,7 @@ set -ev export UNATTENDED="--assume-yes" export MAKE_PARALLEL="-j $(nproc)" -export PDF2HTMLEX_BRANCH="$(git rev-parse --abbrev-ref HEAD)" +export PDF2HTMLEX_BRANCH=$TRAVIS_BRANCH export PDF2HTMLEX_PREFIX=/usr/local From f3e7e60ed20aa2bd20bbe91f32a805147741ec68 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sat, 6 Jun 2020 07:50:22 +0100 Subject: [PATCH 08/69] corrected docker image creation and added additional build documentation --- buildScripts/Readme.md | 45 ++++++++++++++++++++++++++++++++-- buildScripts/createDockerImage | 6 ++--- 2 files changed, 46 insertions(+), 5 deletions(-) diff --git a/buildScripts/Readme.md b/buildScripts/Readme.md index 091bc4c..15ea06c 100644 --- a/buildScripts/Readme.md +++ b/buildScripts/Readme.md @@ -17,6 +17,8 @@ to use these scripts to build `pdf2htmlEX`. - [TL;DR ...](#tldr-) - [Downloading precompiled versions](#downloading-precompiled-versions-downloads) - [Building yourself](#building-yourself) +- [The problem](#the-problem) +- [Our solution](#our-solution) - [The gory details ...](#the-gory-details-) - [Top-level scripts](#top-level-scripts) - [Individual steps](#individual-steps) @@ -50,8 +52,8 @@ For most users, you probably really want to simply download one of the This will work on any [Debian](https://www.debian.org/) based and most recent Windows 10 machines. - Experienced users of Linux, may be able to repackage this `*.deb` for - use with their favourite package management tool. + Experienced users of Linux, may be able to repackage the `*.deb` we + provide for use with their favourite package management tool. ### Building yourself @@ -73,6 +75,45 @@ installing `pdf2htmlEX` into /usr/local/bin. [Debian](https://www.debian.org/) based distribution. such as [Ubuntu](https://ubuntu.com/), [Linux Mint](https://linuxmint.com/), etc. +## The problem + +To provide its full functionality, the `pdf2htmlEX` sources make direct +use of source code and unexposed methods from both the Poppler and +FontForge projects. Unfortunately the source code in the Poppler and +FontForge projects that the `pdf2htmlEX` uses changes regularly. + +This means that the `pdf2htmlEX` souce code *must* be updated regularly to +match *specific releases* of both Poppler and FontForge. + +Unfortunately, the installed versions of both Poppler and FontForge in +most Linux distributions, lag the official releases of both of these +projects. Even worse few distributions install the same versions. + +This means that it is nearly impossible for the `pdf2htmlEX` 'predict' +which version of Poppler or FontForge will be installed on a given user's +machine. + +## Our solution + +While we *could* keep multiple versions of the `pdf2htmlEX` source code, +each version matched to a particular distribution's installed versions of +Poppler and FontForge, this would be a logistic and testing 'nightmare'. + +Instead, when building `pdf2htmlEX`, we download specific versions of both +the Poppler and FontForge sources (usually the most recent), and then +compile *static* versions of the Poppler and FontForge libraries which are +then *statically* linked into the `pdf2htmlEX` binary. + +This means that the `pdf2htmlEX` binary is completely independent of any +locally installed versions of either Poppler or FontForge. + +However, to get the matched versions of Poppler and FontForge and then +compile them statically, *our* build process becomes much more complex +than a "simple", `configure, make, make install` cycle. + +Hence this directory has a large number of bash scripts each of which +automate one simple step in the overall our build process. + ## The gory details ... The bash scripts in this directory automate the download, build, install, diff --git a/buildScripts/createDockerImage b/buildScripts/createDockerImage index 8e4e5cf..0b4fd76 100755 --- a/buildScripts/createDockerImage +++ b/buildScripts/createDockerImage @@ -61,9 +61,9 @@ if [ -x "$(which docker)" ]; then export DOCKER_NAME="$DOCKER_USERNAME/pdf2htmlex:$PDF2HTMLEX_BRANCH-$BUILD_TIME" - echo "export DOCKER_FROM=\"$DOCKER_FROM\"" >> buildScripts/reSourceVersionEnvs - echo "export DOCKER_USERNAME=\"$DOCKER_USERNAME\"" >> buildScripts/reSourceVersionEnvs - echo "export DOCKER_NAME=\"$DOCKER_NAME\"" >> buildScripts/reSourceVersionEnvs + echo "export DOCKER_FROM=\"$DOCKER_FROM\"" >> ../../buildScripts/reSourceVersionEnvs + echo "export DOCKER_USERNAME=\"$DOCKER_USERNAME\"" >> ../../buildScripts/reSourceVersionEnvs + echo "export DOCKER_NAME=\"$DOCKER_NAME\"" >> ../../buildScripts/reSourceVersionEnvs delete_blacklisted From 74170383081c8512a739c73a82430c3aab9a7b38 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sat, 6 Jun 2020 08:18:44 +0100 Subject: [PATCH 09/69] added upload of deb pakcage to github --- buildScripts/uploadGitHubRelease | 1 + 1 file changed, 1 insertion(+) diff --git a/buildScripts/uploadGitHubRelease b/buildScripts/uploadGitHubRelease index b0d23bb..699cfde 100755 --- a/buildScripts/uploadGitHubRelease +++ b/buildScripts/uploadGitHubRelease @@ -79,4 +79,5 @@ uploadAnAsset $upload_url "appImageName.txt" "text/plain" uploadAnAsset $upload_url "buildInfo.sh" "text/plain" uploadAnAsset $upload_url "dockerImageName.txt" "text/plain" uploadAnAsset $upload_url $APPIMAGE_NAME "application/zip" +uploadAnAsset $upload_url $DPKG_NAME "application/x-debian-package" From 6b11b0133ed300f50343f9216ce7162813d76970 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sat, 6 Jun 2020 08:21:31 +0100 Subject: [PATCH 10/69] added upload of deb pakcage to githubTGclwBM --- buildScripts/uploadGitHubRelease | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/buildScripts/uploadGitHubRelease b/buildScripts/uploadGitHubRelease index 699cfde..1f35245 100755 --- a/buildScripts/uploadGitHubRelease +++ b/buildScripts/uploadGitHubRelease @@ -69,15 +69,17 @@ echo " password $GITHUB_TOKEN" >> $HOME/.netrc echo $APPIMAGE_NAME > appImageName.txt echo $DOCKER_NAME > dockerImageName.txt +echo $DPKG_NAME > debianArchiveName.txt deleteReleaseByTag "$TRAVIS_REPO_SLUG" "continuous" createNewRelease "$TRAVIS_REPO_SLUG" "continuous" "Latest release" \ ../buildScripts/uploadGitHubReleaseMessage -uploadAnAsset $upload_url "appImageName.txt" "text/plain" -uploadAnAsset $upload_url "buildInfo.sh" "text/plain" -uploadAnAsset $upload_url "dockerImageName.txt" "text/plain" -uploadAnAsset $upload_url $APPIMAGE_NAME "application/zip" -uploadAnAsset $upload_url $DPKG_NAME "application/x-debian-package" +uploadAnAsset $upload_url "appImageName.txt" "text/plain" +uploadAnAsset $upload_url "buildInfo.sh" "text/plain" +uploadAnAsset $upload_url "dockerImageName.txt" "text/plain" +uploadAnAsset $upload_url "debianArchiveName.txt" "text/plain" +uploadAnAsset $upload_url $APPIMAGE_NAME "application/zip" +uploadAnAsset $upload_url $DPKG_NAME "application/x-debian-package" From d456ae4412a2b352db082473945864f16fd903c3 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sat, 6 Jun 2020 09:00:07 +0100 Subject: [PATCH 11/69] start multiple releases via travis --- .travis.yml | 10 ++++++++-- buildScripts/reportEnvs | 2 ++ buildScripts/uploadGitHubRelease | 4 ++-- 3 files changed, 12 insertions(+), 4 deletions(-) diff --git a/.travis.yml b/.travis.yml index c8aaee5..8ae483e 100644 --- a/.travis.yml +++ b/.travis.yml @@ -17,14 +17,20 @@ branches: jobs: fast_finish: true include: - - stage: build-linux - os: linux + - os: linux dist: bionic env: DOCKER_FROM="ubuntu:18.04" services: - docker script: - ./buildScripts/travisLinuxDoItAll + - os: linux + dist: xenial + env: DOCKER_FROM="ubuntu:16.04" + services: + - docker + script: + - ./buildScripts/travisLinuxDoItAll #deploy: # provider: releases diff --git a/buildScripts/reportEnvs b/buildScripts/reportEnvs index d49fedf..81c5efc 100755 --- a/buildScripts/reportEnvs +++ b/buildScripts/reportEnvs @@ -3,6 +3,7 @@ # This bash script reports the more important TRAVIS environment variables echo "TravisCI env:" +echo " dist: [$TRAVIS_DIST]" echo " tag: [$TRAVIS_TAG]" echo " branch: [$TRAVIS_BRANCH]" echo " commit: [$TRAVIS_COMMIT]" @@ -26,4 +27,5 @@ echo " build time: [$BUILD_TIME]" echo " docker from: [$DOCKER_FROM]" echo " docker name: [$DOCKER_NAME]" echo " appImage name: [$APPIMAGE_NAME]" +echo " dpkg name: [$DPKG_NAME]" echo "uploadTool suffix: [$UPLOADTOOL_SUFFIX]" diff --git a/buildScripts/uploadGitHubRelease b/buildScripts/uploadGitHubRelease index 1f35245..c6beede 100755 --- a/buildScripts/uploadGitHubRelease +++ b/buildScripts/uploadGitHubRelease @@ -71,9 +71,9 @@ echo $APPIMAGE_NAME > appImageName.txt echo $DOCKER_NAME > dockerImageName.txt echo $DPKG_NAME > debianArchiveName.txt -deleteReleaseByTag "$TRAVIS_REPO_SLUG" "continuous" +deleteReleaseByTag "$TRAVIS_REPO_SLUG" "continuous-$TRAVIS_DIST" -createNewRelease "$TRAVIS_REPO_SLUG" "continuous" "Latest release" \ +createNewRelease "$TRAVIS_REPO_SLUG" "continuous-$TRAVIS_DIST" "Latest release" \ ../buildScripts/uploadGitHubReleaseMessage uploadAnAsset $upload_url "appImageName.txt" "text/plain" From 6028a9d2802e40aa2556e7de20af7452fc86553e Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sat, 6 Jun 2020 09:15:01 +0100 Subject: [PATCH 12/69] enable releases on multiple ubuntu distributions --- buildScripts/createAppImage | 4 ++-- buildScripts/createDebianPackage | 6 ++---- buildScripts/reportEnvs | 1 + buildScripts/uploadGitHubRelease | 4 ++-- buildScripts/versionEnvs | 7 ++++++- 5 files changed, 13 insertions(+), 9 deletions(-) diff --git a/buildScripts/createAppImage b/buildScripts/createAppImage index 87f99f3..6e9ba65 100755 --- a/buildScripts/createAppImage +++ b/buildScripts/createAppImage @@ -14,8 +14,8 @@ set -ev # For appimage output plugin # -export VERSION="$PDF2HTMLEX_BRANCH-$BUILD_TIME" -export APPIMAGE_NAME="pdf2htmlEX-$PDF2HTMLEX_BRANCH-$BUILD_TIME-$MACHINE_ARCH.AppImage" +#export VERSION="$PDF2HTMLEX_BRANCH-$BUILD_TIME" +export APPIMAGE_NAME="pdf2htmlEX-$PDF2HTMLEX_BRANCH-$BUILD_TIME-$BUILD_DIST-$MACHINE_ARCH.AppImage" echo "export APPIMAGE_NAME=\"$APPIMAGE_NAME\"" >> buildScripts/reSourceVersionEnvs diff --git a/buildScripts/createDebianPackage b/buildScripts/createDebianPackage index 47c9d8d..6ad501a 100755 --- a/buildScripts/createDebianPackage +++ b/buildScripts/createDebianPackage @@ -10,11 +10,9 @@ echo "CREATING pdf2htmlEX (binary) Debian package" echo "-------------------------------------------------------------------" echo "" -source /etc/lsb-release - set -ev -export DPKG_NAME="pdf2htmlEX-$PDF2HTMLEX_BRANCH-$BUILD_TIME-$MACHINE_ARCH-$DISTRIB_CODENAME.deb" +export DPKG_NAME="pdf2htmlEX-$PDF2HTMLEX_BRANCH-$BUILD_TIME-$MACHINE_ARCH-$BUILD_DIST.deb" echo "export DPKG_NAME=\"$DPKG_NAME\"" >> buildScripts/reSourceVersionEnvs @@ -77,7 +75,7 @@ maintainerValue="$(git config --get user.name) <$(git config --get user.email)>" # echo "Package: pdf2htmlEX" > $controlFile echo "Version: 0:0.$versionValue-0" >> $controlFile -echo "Distribution: $DISTRIB_CODENAME" >> $controlFile +echo "Distribution: $BUILD_DIST" >> $controlFile echo "Architecture: $architectureValue" >> $controlFile echo "Section: universe/web" >> $controlFile echo "Priority: optional" >> $controlFile diff --git a/buildScripts/reportEnvs b/buildScripts/reportEnvs index 81c5efc..48d1538 100755 --- a/buildScripts/reportEnvs +++ b/buildScripts/reportEnvs @@ -14,6 +14,7 @@ echo "Build env:" echo " prefix: [$PDF2HTMLEX_PREFIX]" echo " unattended: [$UNATTENDED]" echo " make parallel: [$MAKE_PARALLEL]" +echo " distribution: [$BULD_DIST]" source ./buildScripts/reSourceVersionEnvs echo "pdf2htmlEX env:" diff --git a/buildScripts/uploadGitHubRelease b/buildScripts/uploadGitHubRelease index c6beede..a7cbab5 100755 --- a/buildScripts/uploadGitHubRelease +++ b/buildScripts/uploadGitHubRelease @@ -71,9 +71,9 @@ echo $APPIMAGE_NAME > appImageName.txt echo $DOCKER_NAME > dockerImageName.txt echo $DPKG_NAME > debianArchiveName.txt -deleteReleaseByTag "$TRAVIS_REPO_SLUG" "continuous-$TRAVIS_DIST" +deleteReleaseByTag "$TRAVIS_REPO_SLUG" "continuous-$BUILD_DIST" -createNewRelease "$TRAVIS_REPO_SLUG" "continuous-$TRAVIS_DIST" "Latest release" \ +createNewRelease "$TRAVIS_REPO_SLUG" "continuous-$BUILD_DIST" "Latest release" \ ../buildScripts/uploadGitHubReleaseMessage uploadAnAsset $upload_url "appImageName.txt" "text/plain" diff --git a/buildScripts/versionEnvs b/buildScripts/versionEnvs index 11a4f69..fbc708d 100755 --- a/buildScripts/versionEnvs +++ b/buildScripts/versionEnvs @@ -40,13 +40,18 @@ if [ -z "$PDF2HTMLEX_BRANCH" ]; then fi fi -export MACHINE_ARCH="$(uname -m)" +source /etc/lsb-release + +export BUILD_DIST=$DISTRIB_CODENAME export BUILD_TIME="$(date +%Y_%m_%d-%H_%M_%S)" +export MACHINE_ARCH="$(uname -m)" + echo "export POPPLER_VERSION=\"$POPPLER_VERSION\"" > buildScripts/reSourceVersionEnvs echo "export FONTFORGE_VERSION=\"$FONTFORGE_VERSION\"" >> buildScripts/reSourceVersionEnvs echo "export PDF2HTMLEX_BRANCH=\"$PDF2HTMLEX_BRANCH\"" >> buildScripts/reSourceVersionEnvs echo "export MACHINE_ARCH=\"$MACHINE_ARCH\"" >> buildScripts/reSourceVersionEnvs +echo "export BUILD_DIST=\"$BUILD_DIST\"" >> buildScripts/reSourceVersionEnvs echo "export BUILD_TIME=\"$BUILD_TIME\"" >> buildScripts/reSourceVersionEnvs echo "export PDF2HTMLEX_PREFIX=\"$PDF2HTMLEX_PREFIX\"" >> buildScripts/reSourceVersionEnvs From 1cb016fd14fc68c23514aa620daf14019acff900 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sat, 6 Jun 2020 09:22:08 +0100 Subject: [PATCH 13/69] enable releases on multiple ubuntu distributions reattempt travis --- .travis.yml | 7 ------- 1 file changed, 7 deletions(-) diff --git a/.travis.yml b/.travis.yml index 8ae483e..97388df 100644 --- a/.travis.yml +++ b/.travis.yml @@ -24,13 +24,6 @@ jobs: - docker script: - ./buildScripts/travisLinuxDoItAll - - os: linux - dist: xenial - env: DOCKER_FROM="ubuntu:16.04" - services: - - docker - script: - - ./buildScripts/travisLinuxDoItAll #deploy: # provider: releases From 5ddc84e7f86c272a960182855d382a6f983de295 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sat, 6 Jun 2020 09:25:50 +0100 Subject: [PATCH 14/69] fixed reportEnvs --- buildScripts/reportEnvs | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/buildScripts/reportEnvs b/buildScripts/reportEnvs index 48d1538..331a796 100755 --- a/buildScripts/reportEnvs +++ b/buildScripts/reportEnvs @@ -14,7 +14,8 @@ echo "Build env:" echo " prefix: [$PDF2HTMLEX_PREFIX]" echo " unattended: [$UNATTENDED]" echo " make parallel: [$MAKE_PARALLEL]" -echo " distribution: [$BULD_DIST]" +echo " distribution: [$BUILD_DIST]" + source ./buildScripts/reSourceVersionEnvs echo "pdf2htmlEX env:" From 5165d5d6e7fba2e24c02e8d6f23e6cc55e63f92a Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sat, 6 Jun 2020 09:29:55 +0100 Subject: [PATCH 15/69] fixed versionEnvs --- buildScripts/versionEnvs | 2 ++ 1 file changed, 2 insertions(+) diff --git a/buildScripts/versionEnvs b/buildScripts/versionEnvs index fbc708d..0bc662a 100755 --- a/buildScripts/versionEnvs +++ b/buildScripts/versionEnvs @@ -40,6 +40,8 @@ if [ -z "$PDF2HTMLEX_BRANCH" ]; then fi fi +cat /etc/lsb-release + source /etc/lsb-release export BUILD_DIST=$DISTRIB_CODENAME From cee878e667219834251e9586d2b508ea7e759b4d Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sat, 6 Jun 2020 09:34:34 +0100 Subject: [PATCH 16/69] test travis --- buildScripts/versionEnvs | 2 ++ 1 file changed, 2 insertions(+) diff --git a/buildScripts/versionEnvs b/buildScripts/versionEnvs index 0bc662a..5bd12be 100755 --- a/buildScripts/versionEnvs +++ b/buildScripts/versionEnvs @@ -44,6 +44,8 @@ cat /etc/lsb-release source /etc/lsb-release +echo "DISTRIB_CODENAME = $DISTRIB_CODENAME" + export BUILD_DIST=$DISTRIB_CODENAME export BUILD_TIME="$(date +%Y_%m_%d-%H_%M_%S)" From c69a56f41e24453259302bdbcd18a4e173d154f7 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sat, 6 Jun 2020 09:37:55 +0100 Subject: [PATCH 17/69] test travis 2 --- buildScripts/versionEnvs | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/buildScripts/versionEnvs b/buildScripts/versionEnvs index 5bd12be..38b0ff3 100755 --- a/buildScripts/versionEnvs +++ b/buildScripts/versionEnvs @@ -48,6 +48,8 @@ echo "DISTRIB_CODENAME = $DISTRIB_CODENAME" export BUILD_DIST=$DISTRIB_CODENAME +echo "BUILD_DIST = $BUILD_DIST" + export BUILD_TIME="$(date +%Y_%m_%d-%H_%M_%S)" export MACHINE_ARCH="$(uname -m)" @@ -59,3 +61,5 @@ echo "export MACHINE_ARCH=\"$MACHINE_ARCH\"" >> buildScripts/reSourceV echo "export BUILD_DIST=\"$BUILD_DIST\"" >> buildScripts/reSourceVersionEnvs echo "export BUILD_TIME=\"$BUILD_TIME\"" >> buildScripts/reSourceVersionEnvs echo "export PDF2HTMLEX_PREFIX=\"$PDF2HTMLEX_PREFIX\"" >> buildScripts/reSourceVersionEnvs + +cat buildScripts/reSourceVersionEnvs From 76bd542ae5aba714da89f9dd41ddecad98a30360 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sat, 6 Jun 2020 09:40:24 +0100 Subject: [PATCH 18/69] test travis 3 --- buildScripts/reportEnvs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/buildScripts/reportEnvs b/buildScripts/reportEnvs index 331a796..7630fc1 100755 --- a/buildScripts/reportEnvs +++ b/buildScripts/reportEnvs @@ -2,6 +2,8 @@ # This bash script reports the more important TRAVIS environment variables +source ./buildScripts/reSourceVersionEnvs + echo "TravisCI env:" echo " dist: [$TRAVIS_DIST]" echo " tag: [$TRAVIS_TAG]" @@ -16,8 +18,6 @@ echo " unattended: [$UNATTENDED]" echo " make parallel: [$MAKE_PARALLEL]" echo " distribution: [$BUILD_DIST]" -source ./buildScripts/reSourceVersionEnvs - echo "pdf2htmlEX env:" echo " poppler: [$POPPLER_VERSION]" echo " fontforge: [$FONTFORGE_VERSION]" From 570421d04af5e33058b680b4f8879c59bd6ae8c9 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sat, 6 Jun 2020 09:59:52 +0100 Subject: [PATCH 19/69] corrected travis release scripts --- buildScripts/uploadGitHubRelease | 2 +- buildScripts/versionEnvs | 6 ------ 2 files changed, 1 insertion(+), 7 deletions(-) diff --git a/buildScripts/uploadGitHubRelease b/buildScripts/uploadGitHubRelease index a7cbab5..0f14170 100755 --- a/buildScripts/uploadGitHubRelease +++ b/buildScripts/uploadGitHubRelease @@ -73,7 +73,7 @@ echo $DPKG_NAME > debianArchiveName.txt deleteReleaseByTag "$TRAVIS_REPO_SLUG" "continuous-$BUILD_DIST" -createNewRelease "$TRAVIS_REPO_SLUG" "continuous-$BUILD_DIST" "Latest release" \ +createNewRelease "$TRAVIS_REPO_SLUG" "continuous-$BUILD_DIST" "Latest $BUILD_DIST release" \ ../buildScripts/uploadGitHubReleaseMessage uploadAnAsset $upload_url "appImageName.txt" "text/plain" diff --git a/buildScripts/versionEnvs b/buildScripts/versionEnvs index 38b0ff3..0bc662a 100755 --- a/buildScripts/versionEnvs +++ b/buildScripts/versionEnvs @@ -44,12 +44,8 @@ cat /etc/lsb-release source /etc/lsb-release -echo "DISTRIB_CODENAME = $DISTRIB_CODENAME" - export BUILD_DIST=$DISTRIB_CODENAME -echo "BUILD_DIST = $BUILD_DIST" - export BUILD_TIME="$(date +%Y_%m_%d-%H_%M_%S)" export MACHINE_ARCH="$(uname -m)" @@ -61,5 +57,3 @@ echo "export MACHINE_ARCH=\"$MACHINE_ARCH\"" >> buildScripts/reSourceV echo "export BUILD_DIST=\"$BUILD_DIST\"" >> buildScripts/reSourceVersionEnvs echo "export BUILD_TIME=\"$BUILD_TIME\"" >> buildScripts/reSourceVersionEnvs echo "export PDF2HTMLEX_PREFIX=\"$PDF2HTMLEX_PREFIX\"" >> buildScripts/reSourceVersionEnvs - -cat buildScripts/reSourceVersionEnvs From a3b712d8931ccb3bce6eab3b973c2b280492b755 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sat, 6 Jun 2020 10:39:36 +0000 Subject: [PATCH 20/69] added creation and upload of test results --- .travis.yml | 13 ++----------- buildScripts/runTests | 20 ++++++++++++++++++-- buildScripts/uploadGitHubRelease | 9 ++++++--- pdf2htmlEX/CMakeLists.txt | 1 + pdf2htmlEX/test/browser_tests.py | 14 ++++++++++++++ pdf2htmlEX/test/installAutomaticTestSoftware | 4 ++++ pdf2htmlEX/test/runLocalBrowserTests | 8 +++++--- pdf2htmlEX/test/test.py.in | 1 + 8 files changed, 51 insertions(+), 19 deletions(-) diff --git a/.travis.yml b/.travis.yml index 97388df..24e0ff6 100644 --- a/.travis.yml +++ b/.travis.yml @@ -22,16 +22,7 @@ jobs: env: DOCKER_FROM="ubuntu:18.04" services: - docker + addons: + firefox: latest script: - ./buildScripts/travisLinuxDoItAll - -#deploy: -# provider: releases -# api_key: -# secure: ALWKAkgeLSPS4ULyTtxU50ZmSICEafCirCQ1YmuVTqe9nrc8E9Jm+4QaBlab5k6v90EpzNvZBnMOiT8cT1Wp2lqJzcGdGMcG49mhRvirdYhXgWuHYnygVgkxEhgejXcQaEiaGPDV3Y1k9MuiAVr8ApjUvIMlY4Hg1y0r7/rqR2fYmen+f+6a87bp5PptwakC9bfoYb92651d0TbugFfKT6TGKlE2jGlQ6On1alzLQ2DXA8coUjwpp4NcDD8OhnYGN/VZWkqckaBrMrHwjJ5FTfy5MGBOqCLzNix8sCGLEeCKkKC8J0nniNJ7+KOu9LJ2uIezKYfaUOJV0kik7F9ErvxjbfSRieako2yaSOR8X39aicTtpbyoXPRSKDU4WYX56vt4Dx7aNrYtP5b/UwWGXZa9MK7rNGJ+hiMDO1AzuyXirzR4RaJXCyC8n5W9ahBZBHCMqd5gMg5sMGPNj3Rgt34+R+nGYssum5wvr+6IKNl57sIoZFCgqviBrw2gJOCqlgOcEk/4ZSehWnXPCk6Gzxvkqkt/dmp8pJLNyvMaWcEZ6CXNHDU8MZS7bASo8Z7eAYvmX0MMjVty8J47expVaJ8M3OFHu6lE6tU8wwfd+Igr2TlWcixqaa3hDMgS+ghwdiZbYuMdR1AyZnEqK3xwvXXPPXrqRsB9xHtczLibkjY= -# file: pdf2htmlEX-newBuildSystem-x86_64.AppImage -# skip_cleanup: true -# on: -# tags: true -# repo: stephengaito/pdf2htmlEX - diff --git a/buildScripts/runTests b/buildScripts/runTests index 4a1f355..a218cca 100755 --- a/buildScripts/runTests +++ b/buildScripts/runTests @@ -1,5 +1,16 @@ #!/bin/bash +source ./buildScripts/reSourceVersionEnvs + +echo "" +echo "-------------------------------------------------------------------" +echo "RUNNING pdf2htmlEX tests" +echo "-------------------------------------------------------------------" +echo "" + +export TEST_RESULTS_NAME="pdf2htmlEX-$PDF2HTMLEX_BRANCH-$BUILD_TIME-$BUILD_DIST-$MACHINE_ARCH-testResults.zip" +echo "export TEST_RESULTS_NAME=\"$TEST_RESULTS_NAME\"" >> buildScripts/reSourceVersionEnvs + set -ev # This bash script runs the pdf2htmlEX tests @@ -8,9 +19,14 @@ cd pdf2htmlEX/test # The following is only needed for the local browser tests # -#./installAutomaticTestSoftware +./installAutomaticTestSoftware ./runLocalTests -#./runLocalBrowserTests +./runLocalBrowserTests +cd ../.. + +mkdir -p imageBuild + +zip -r imageBuild/$TEST_RESULTS_NAME /tmp/pdf2htmlEX/html /tmp/pdf2htmlEX/png diff --git a/buildScripts/uploadGitHubRelease b/buildScripts/uploadGitHubRelease index 0f14170..a918ab7 100755 --- a/buildScripts/uploadGitHubRelease +++ b/buildScripts/uploadGitHubRelease @@ -67,9 +67,10 @@ echo "machine uploads.github.com" >> $HOME/.netrc echo " login $GITHUB_USERNAME" >> $HOME/.netrc echo " password $GITHUB_TOKEN" >> $HOME/.netrc -echo $APPIMAGE_NAME > appImageName.txt -echo $DOCKER_NAME > dockerImageName.txt -echo $DPKG_NAME > debianArchiveName.txt +echo $APPIMAGE_NAME > appImageName.txt +echo $DOCKER_NAME > dockerImageName.txt +echo $DPKG_NAME > debianArchiveName.txt +echo $TEST_RESULTS_NAME > testResultsName.txt deleteReleaseByTag "$TRAVIS_REPO_SLUG" "continuous-$BUILD_DIST" @@ -77,9 +78,11 @@ createNewRelease "$TRAVIS_REPO_SLUG" "continuous-$BUILD_DIST" "Latest $BUILD_D ../buildScripts/uploadGitHubReleaseMessage uploadAnAsset $upload_url "appImageName.txt" "text/plain" +uploadAnAsset $upload_url "testResultsName.txt" "text/plain" uploadAnAsset $upload_url "buildInfo.sh" "text/plain" uploadAnAsset $upload_url "dockerImageName.txt" "text/plain" uploadAnAsset $upload_url "debianArchiveName.txt" "text/plain" uploadAnAsset $upload_url $APPIMAGE_NAME "application/zip" +uploadAnAsset $upload_url $TEST_RESULTS_NAME "application/zip" uploadAnAsset $upload_url $DPKG_NAME "application/x-debian-package" diff --git a/pdf2htmlEX/CMakeLists.txt b/pdf2htmlEX/CMakeLists.txt index 691ead1..ab3c61a 100644 --- a/pdf2htmlEX/CMakeLists.txt +++ b/pdf2htmlEX/CMakeLists.txt @@ -262,6 +262,7 @@ set(PDF2HTMLEX_TMPDIR /tmp/pdf2htmlEX/tmp) set(PDF2HTMLEX_DATDIR /tmp/pdf2htmlEX/dat) set(PDF2HTMLEX_PNGDIR /tmp/pdf2htmlEX/png) set(PDF2HTMLEX_OUTDIR /tmp/pdf2htmlEX/out) +set(PDF2HTMLEX_HTMDIR /tmp/pdf2htmlEX/html) file(MAKE_DIRECTORY ${PDF2HTMLEX_TMPDIR}) file(MAKE_DIRECTORY ${PDF2HTMLEX_DATDIR}) file(MAKE_DIRECTORY ${PDF2HTMLEX_PNGDIR}) diff --git a/pdf2htmlEX/test/browser_tests.py b/pdf2htmlEX/test/browser_tests.py index 74634fa..93906ad 100644 --- a/pdf2htmlEX/test/browser_tests.py +++ b/pdf2htmlEX/test/browser_tests.py @@ -51,6 +51,20 @@ class BrowserTests(Common): shutil.copytree(self.OUTDIR, ref_htmlfolder) return + # keep a record of the HTML files (and any differences) + # for later reporting of test results + # + try: + os.makedirs(self.HTMDIR, 0o755, True) + outHtmlFile = os.path.join(self.HTMDIR, basefilename+'.out.html') + refHtmlFile = os.path.join(self.HTMDIR, basefilename+'.ref.html') + difHtmlFile = os.path.join(self.HTMDIR, basefilename+'.diff.html') + shutil.copy(out_htmlfilename, outHtmlFile) + shutil.copy(ref_htmlfilename, refHtmlFile) + os.system("diff "+outHtmlFile+" "+refHtmlFile+" > "+difHtmlFile+" 2>&1") + except: + pass + pngfilename_out = os.path.join(self.PNGDIR, basefilename + '.out.png') self.generate_image(out_htmlfilename, pngfilename_out) out_img = Image.open(pngfilename_out).convert('RGB') diff --git a/pdf2htmlEX/test/installAutomaticTestSoftware b/pdf2htmlEX/test/installAutomaticTestSoftware index 891dcd5..032c2d0 100755 --- a/pdf2htmlEX/test/installAutomaticTestSoftware +++ b/pdf2htmlEX/test/installAutomaticTestSoftware @@ -1,5 +1,7 @@ #!/bin/bash +set -ev + # This bash script installs all local software required to run the # pdf2htmlEX tests @@ -8,6 +10,8 @@ export DEBIAN_FRONTEND=noninteractive # Start by making sure all required apt packages exist # sudo apt -y install \ + diffutils \ + zip \ python3 \ python3-pip \ xvfb \ diff --git a/pdf2htmlEX/test/runLocalBrowserTests b/pdf2htmlEX/test/runLocalBrowserTests index 4d122c7..f65af7b 100755 --- a/pdf2htmlEX/test/runLocalBrowserTests +++ b/pdf2htmlEX/test/runLocalBrowserTests @@ -1,7 +1,5 @@ #!/bin/bash -set -ev - # This bash script runs the local browser tests # We start by running a virtual frame buffer as display 99.0 @@ -50,4 +48,8 @@ ps -ef | grep -v grep | grep Xvfb echo "-----------------------------------------------------------" echo "" -exit $returnCode +if [ -z "$TRAVIS_DIST"] ; then + exit $returnCode +fi + +exit 0 diff --git a/pdf2htmlEX/test/test.py.in b/pdf2htmlEX/test/test.py.in index 495c4c0..16d619e 100755 --- a/pdf2htmlEX/test/test.py.in +++ b/pdf2htmlEX/test/test.py.in @@ -28,6 +28,7 @@ class Common(object): PNGDIR = "@PDF2HTMLEX_PNGDIR@" DATDIR = "@PDF2HTMLEX_DATDIR@" OUTDIR = "@PDF2HTMLEX_OUTDIR@" + HTMDIR = "@PDF2HTMLEX_HTMDIR@" def setUp(self): # filter manifest From 8f86e21b8f47f94a906fed41d01df7bc594cf943 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sat, 6 Jun 2020 11:52:52 +0100 Subject: [PATCH 21/69] test travis --- buildScripts/uploadGitHubRelease | 2 ++ 1 file changed, 2 insertions(+) diff --git a/buildScripts/uploadGitHubRelease b/buildScripts/uploadGitHubRelease index a918ab7..ad4d690 100755 --- a/buildScripts/uploadGitHubRelease +++ b/buildScripts/uploadGitHubRelease @@ -60,6 +60,8 @@ fi cd imageBuild +ls -la + echo "machine api.github.com" > $HOME/.netrc echo " login $GITHUB_USERNAME" >> $HOME/.netrc echo " password $GITHUB_TOKEN" >> $HOME/.netrc From aaa7f53c587790ac399998798afe3689b5f43bce Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sat, 6 Jun 2020 12:31:17 +0100 Subject: [PATCH 22/69] corrected appimage creation --- buildScripts/createAppImage | 3 +++ 1 file changed, 3 insertions(+) diff --git a/buildScripts/createAppImage b/buildScripts/createAppImage index 6e9ba65..0526ab1 100755 --- a/buildScripts/createAppImage +++ b/buildScripts/createAppImage @@ -59,6 +59,9 @@ if [ -f /.dockerenv ] ; then LINUX_DEPLOY_APP_IMAGE=squashfs-root/AppRun fi +export VERBOSE=1 +export OUTPUT=$APPIMAGE_NAME + ./$LINUX_DEPLOY_APP_IMAGE \ -e appDir/$PDF2HTMLEX_PREFIX/bin/pdf2htmlEX \ --create-desktop-file \ From 87bc1ba525c808ea2a96492ea6ebd9a88c5357d9 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sat, 6 Jun 2020 12:53:21 +0100 Subject: [PATCH 23/69] updated release message --- buildScripts/uploadGitHubReleaseMessage | 33 ++++++++++++++++++++++--- 1 file changed, 30 insertions(+), 3 deletions(-) diff --git a/buildScripts/uploadGitHubReleaseMessage b/buildScripts/uploadGitHubReleaseMessage index 9c83d0d..474f98e 100644 --- a/buildScripts/uploadGitHubReleaseMessage +++ b/buildScripts/uploadGitHubReleaseMessage @@ -1,6 +1,33 @@ -This is the development/testing release of the pdf2htmlEX AppImage. +This is the most recent release of the pdf2htmlEX project. -You can download the AppImage and 'just run it'. +We release three types of binary objects: -See: https://appimage.org/ for details +1. The file *.AppImage on this release page is an AppImage + You can download the AppImage, make it executable and + then 'just run it'. + + See: https://appimage.org/ for details + +2. The file *.deb is a Debian Archive + + You can download the Debian Archive and + install it locally using: + + sudo apt install <> + +3. You can alternatively get pdf2htmlEX Docker images + from Docker Hub in either of the repositories: + + pdf2htmlEX/pdf2htmlEX (offical release) + + or + + stephengaito/pdf2htmlEX (development releases) + +We also release a zip archive of the browser test results +in the *-testResults.zip file. + +Finally, the various *.txt *buildInfo.sh files have well known +(unchanging) names which can be used by automatic scripts to identifiy the +current full names of the binary artifacts. From d5fd346ba0d3981478798426a979d0bdcd904d72 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sat, 6 Jun 2020 13:09:54 +0100 Subject: [PATCH 24/69] corrected documentation --- buildScripts/Readme.md | 16 ++++------- buildScripts/createDebianPackage | 2 +- buildScripts/uploadGitHubReleaseMessage | 22 ++++++++------- testScripts/doItAll | 36 ------------------------- testScripts/doItAllTravis | 32 ---------------------- 5 files changed, 18 insertions(+), 90 deletions(-) delete mode 100755 testScripts/doItAll delete mode 100755 testScripts/doItAllTravis diff --git a/buildScripts/Readme.md b/buildScripts/Readme.md index 15ea06c..082cfd8 100644 --- a/buildScripts/Readme.md +++ b/buildScripts/Readme.md @@ -29,7 +29,7 @@ to use these scripts to build `pdf2htmlEX`. ## TL;DR ... -### Downloading precompiled versions {#Downloads} +### Downloading precompiled versions For most users, you probably really want to simply download one of the [precompiled versions of @@ -89,9 +89,9 @@ Unfortunately, the installed versions of both Poppler and FontForge in most Linux distributions, lag the official releases of both of these projects. Even worse few distributions install the same versions. -This means that it is nearly impossible for the `pdf2htmlEX` 'predict' -which version of Poppler or FontForge will be installed on a given user's -machine. +This means that it is nearly impossible for the `pdf2htmlEX` code to +'predict' which version of Poppler or FontForge will be installed on a +given user's machine. ## Our solution @@ -162,8 +162,7 @@ Typically, most users, will run one of the following "top-level" scripts: Following successful `buildInstallLocally`, `createImages` and `runTests`, this will automate the upload of the various artefacts to - the `pdf2htmlEX` releases page, LaunchPad ppa and docker hub - repository. + the `pdf2htmlEX` releases page, and docker hub repository. **Note** that this step requires the user to enter passwords for each of the respective services. *Most* users will not need (or be able) to @@ -241,11 +240,6 @@ Typically, most users, will run one of the following "top-level" scripts: `pdf2htmlEX/pdf2htmlEX/test` directory. See the `pdf2htmlEX/pdf2htmlEx/test` directory's Readme file for details. -- **`uploadDebianArchive`**: Upload the `pdf2htmlEX` Debian archive to - both the GitHub releases page, as well as the LaunchPad ppa. - - This script has not yet been implemented. - - **`uploadDockerImage`**: Upload the `pdf2htmlEX` Docker image to Docker hub repository associated to the docker hub users specified in the `DOCKER_USERNAME` environement variable. diff --git a/buildScripts/createDebianPackage b/buildScripts/createDebianPackage index 6ad501a..9f600e5 100755 --- a/buildScripts/createDebianPackage +++ b/buildScripts/createDebianPackage @@ -12,7 +12,7 @@ echo "" set -ev -export DPKG_NAME="pdf2htmlEX-$PDF2HTMLEX_BRANCH-$BUILD_TIME-$MACHINE_ARCH-$BUILD_DIST.deb" +export DPKG_NAME="pdf2htmlEX-$PDF2HTMLEX_BRANCH-$BUILD_TIME-$BUILD_DIST-$MACHINE_ARCH.deb" echo "export DPKG_NAME=\"$DPKG_NAME\"" >> buildScripts/reSourceVersionEnvs diff --git a/buildScripts/uploadGitHubReleaseMessage b/buildScripts/uploadGitHubReleaseMessage index 474f98e..0070e94 100644 --- a/buildScripts/uploadGitHubReleaseMessage +++ b/buildScripts/uploadGitHubReleaseMessage @@ -4,20 +4,22 @@ We release three types of binary objects: 1. The file *.AppImage on this release page is an AppImage - You can download the AppImage, make it executable and - then 'just run it'. + You can download the AppImage, make it executable and then 'just run + it'. See: https://appimage.org/ for details 2. The file *.deb is a Debian Archive - You can download the Debian Archive and - install it locally using: + You can download the Debian Archive and install it locally using: sudo apt install <> -3. You can alternatively get pdf2htmlEX Docker images - from Docker Hub in either of the repositories: + While the Debian archive is built on a specific Ubuntu release, it can + usually be used on any more recent Ubuntu releases. + +3. You can alternatively get pdf2htmlEX Docker images from Docker Hub in + either of the repositories: pdf2htmlEX/pdf2htmlEX (offical release) @@ -25,9 +27,9 @@ We release three types of binary objects: stephengaito/pdf2htmlEX (development releases) -We also release a zip archive of the browser test results -in the *-testResults.zip file. +We also release a zip archive of the browser test results in the +*-testResults.zip file. -Finally, the various *.txt *buildInfo.sh files have well known +Finally, the various *.txt and *buildInfo.sh files have well known (unchanging) names which can be used by automatic scripts to identifiy the -current full names of the binary artifacts. +current full names of the corresponding binary artifacts. diff --git a/testScripts/doItAll b/testScripts/doItAll deleted file mode 100755 index a10e143..0000000 --- a/testScripts/doItAll +++ /dev/null @@ -1,36 +0,0 @@ -#!/bin/bash - -# This bash script automates the tests - -mkdir -p testDir - -cd testDir - -wget https://github.com/stephengaito/pdf2htmlEX/releases/download/continuous/appImageName - -export APPIMAGE_NAME=$(cat appImageName) - -wget https://github.com/stephengaito/pdf2htmlEX/releases/download/continuous/$APPIMAGE_NAME - -chmod a+x $APPIMAGE_NAME - -if ! ./$APPIMAGE_NAME --version ; then - echo "COULD NOT run the AppImage" - exit 1 -fi - -if [ -x "$(which docker)" ]; then - - wget https://github.com/stephengaito/pdf2htmlEX/releases/download/continuous/dockerImageName - - export DOCKER_IMAGE_NAME=$(cat dockerImageName) - - if ! docker run $DOCKER_IMAGE_NAME --version ; then - echo "COULD NOT run the Docker Image" - exit 1 - fi - -fi - - - diff --git a/testScripts/doItAllTravis b/testScripts/doItAllTravis deleted file mode 100755 index 695fd75..0000000 --- a/testScripts/doItAllTravis +++ /dev/null @@ -1,32 +0,0 @@ -#!/bin/bash - -# This bash script automates the tests - -wget https://github.com/stephengaito/pdf2htmlEX/releases/download/continuous/appImageName - -export APPIMAGE_NAME=$(cat appImageName) - -wget https://github.com/stephengaito/pdf2htmlEX/releases/download/continuous/$APPIMAGE_NAME - -chmod a+x $APPIMAGE_NAME - -if ! ./$APPIMAGE_NAME --version ; then - echo "COULD NOT run the AppImage" - exit 1 -fi - -if [ -x "$(which docker)" ]; then - - wget https://github.com/stephengaito/pdf2htmlEX/releases/download/continuous/dockerImageName - - export DOCKER_IMAGE_NAME=$(cat dockerImageName) - - if ! docker run $DOCKER_IMAGE_NAME --version ; then - echo "COULD NOT run the Docker Image" - exit 1 - fi - -fi - - - From 174f28a21a70ea1f543d6a5368b96e9ca9a83ab3 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Tue, 9 Jun 2020 12:52:18 +0100 Subject: [PATCH 25/69] updated required build tools --- buildScripts/getBuildTools | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/buildScripts/getBuildTools b/buildScripts/getBuildTools index 41a9b7e..940a00e 100755 --- a/buildScripts/getBuildTools +++ b/buildScripts/getBuildTools @@ -16,7 +16,7 @@ set -ev sudo apt-get update sudo apt-get $UNATTENDED install \ - aptitude \ + wget \ git \ pkg-config \ ruby \ @@ -30,6 +30,5 @@ sudo apt-get $UNATTENDED install \ dpkg-dev \ gettext \ openjdk-8-jre-headless \ - jq \ - tree + jq From d327b43a434e587dae4817af2bf0ec99d13f3106 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Thu, 11 Jun 2020 06:37:23 +0100 Subject: [PATCH 26/69] ensure files copied into AppImage Debian archive and Docker image are readable added sudo as a build tool --- buildScripts/createAppImage | 8 ++++++++ buildScripts/createDebianPackage | 12 ++++++++++++ buildScripts/createDockerImage | 9 +++++++++ buildScripts/getBuildTools | 1 + 4 files changed, 30 insertions(+) diff --git a/buildScripts/createAppImage b/buildScripts/createAppImage index 0526ab1..17c03cf 100755 --- a/buildScripts/createAppImage +++ b/buildScripts/createAppImage @@ -36,6 +36,14 @@ make install \ cd ../imageBuild +# Make sure directories can be traversed by nobody +# +find appDir -type d -exec chmod 755 {} \; +# +# Make sure files can be read by nobody +# +find appDir -type f -exec chmod 644 {} \; + # force libfontconfig into AppImage (linuxdeploy blacklists libfontconfig) # (turned off since libfontconfig needs to be matched to the underlying # OS's collection of fonts and /etc/fonts configuration files) diff --git a/buildScripts/createDebianPackage b/buildScripts/createDebianPackage index 9f600e5..4f7fd34 100755 --- a/buildScripts/createDebianPackage +++ b/buildScripts/createDebianPackage @@ -94,6 +94,18 @@ touch $conffilesFile # cd imageBuild # +# Make sure directories can be traversed by nobody +# +find debianDir -type d -exec chmod 755 {} \; +# +# Make sure files can be read by nobody +# +find debianDir -type f -exec chmod 644 {} \; +# +# Make sure root:root owns all files +# sudo chown -R root:root debianDir # +# Build the package +# dpkg --build debianDir $DPKG_NAME diff --git a/buildScripts/createDockerImage b/buildScripts/createDockerImage index 0b4fd76..b236682 100755 --- a/buildScripts/createDockerImage +++ b/buildScripts/createDockerImage @@ -35,6 +35,15 @@ cd ../imageBuild/dockerDir copy_deps +# Make sure directories can be traversed by nobody +# +find . -type d -exec chmod 755 {} \; +# +# Make sure files can be read by nobody +# +find . -type f -exec chmod 644 {} \; + + # Now. IF we have docker, THEN build the docker image... if [ -x "$(which docker)" ]; then diff --git a/buildScripts/getBuildTools b/buildScripts/getBuildTools index 940a00e..a0e8e35 100755 --- a/buildScripts/getBuildTools +++ b/buildScripts/getBuildTools @@ -16,6 +16,7 @@ set -ev sudo apt-get update sudo apt-get $UNATTENDED install \ + sudo \ wget \ git \ pkg-config \ From c10efe8ecebfc56a91c91a797a8f1025bc047466 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Thu, 11 Jun 2020 09:05:14 +0100 Subject: [PATCH 27/69] updated createAppImage to understand permissions problems --- buildScripts/createAppImage | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/buildScripts/createAppImage b/buildScripts/createAppImage index 17c03cf..8c33d52 100755 --- a/buildScripts/createAppImage +++ b/buildScripts/createAppImage @@ -38,11 +38,11 @@ cd ../imageBuild # Make sure directories can be traversed by nobody # -find appDir -type d -exec chmod 755 {} \; +#find appDir -type d -exec chmod 755 {} \; # # Make sure files can be read by nobody # -find appDir -type f -exec chmod 644 {} \; +#find appDir -type f -exec chmod 644 {} \; # force libfontconfig into AppImage (linuxdeploy blacklists libfontconfig) # (turned off since libfontconfig needs to be matched to the underlying From 501b73b489e33bf098fa1bc8b657f1b4e886039f Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Thu, 11 Jun 2020 09:27:47 +0100 Subject: [PATCH 28/69] exploring create image permissions problems --- buildScripts/createDebianPackage | 4 ++-- buildScripts/createDockerImage | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/buildScripts/createDebianPackage b/buildScripts/createDebianPackage index 4f7fd34..6ddb7b1 100755 --- a/buildScripts/createDebianPackage +++ b/buildScripts/createDebianPackage @@ -96,11 +96,11 @@ cd imageBuild # # Make sure directories can be traversed by nobody # -find debianDir -type d -exec chmod 755 {} \; +#find debianDir -type d -exec chmod 755 {} \; # # Make sure files can be read by nobody # -find debianDir -type f -exec chmod 644 {} \; +#find debianDir -type f -exec chmod 644 {} \; # # Make sure root:root owns all files # diff --git a/buildScripts/createDockerImage b/buildScripts/createDockerImage index b236682..c5f0c9b 100755 --- a/buildScripts/createDockerImage +++ b/buildScripts/createDockerImage @@ -37,11 +37,11 @@ copy_deps # Make sure directories can be traversed by nobody # -find . -type d -exec chmod 755 {} \; +#find . -type d -exec chmod 755 {} \; # # Make sure files can be read by nobody # -find . -type f -exec chmod 644 {} \; +#find . -type f -exec chmod 644 {} \; # Now. IF we have docker, THEN build the docker image... From bf924dca2d2dca05423fb4b7040c181962129456 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Fri, 12 Jun 2020 14:50:49 +0100 Subject: [PATCH 29/69] explicitly split createDockerImage into an inside and an outside docker container parts --- buildScripts/createDockerImage | 93 --------------------------- buildScripts/createDockerImageInside | 46 +++++++++++++ buildScripts/createDockerImageOutside | 62 ++++++++++++++++++ 3 files changed, 108 insertions(+), 93 deletions(-) delete mode 100755 buildScripts/createDockerImage create mode 100755 buildScripts/createDockerImageInside create mode 100755 buildScripts/createDockerImageOutside diff --git a/buildScripts/createDockerImage b/buildScripts/createDockerImage deleted file mode 100755 index c5f0c9b..0000000 --- a/buildScripts/createDockerImage +++ /dev/null @@ -1,93 +0,0 @@ -#!/bin/bash - -# This bash script creates a docker image from an existing pdf2htmlEX - -echo "" -echo "-------------------------------------------------------------------" -echo "CREATING pdf2htmlEX Docker Image" -echo "-------------------------------------------------------------------" -echo "" - -# Collect everything that will be needed... - -source buildScripts/reSourceVersionEnvs - -source buildScripts/dockerFunctions - -set -ev - -mkdir -p imageBuild/dockerDir - -cd pdf2htmlEX/build - -sudo rm -rf install_manifest.txt - -make install DESTDIR=../../imageBuild/dockerDir - -cd ../../poppler-data - -make install \ - prefix=$PDF2HTMLEX_PREFIX \ - datadir=$PDF2HTMLEX_PREFIX/share/pdf2htmlEX \ - DESTDIR=../imageBuild/dockerDir - -cd ../imageBuild/dockerDir - -copy_deps - -# Make sure directories can be traversed by nobody -# -#find . -type d -exec chmod 755 {} \; -# -# Make sure files can be read by nobody -# -#find . -type f -exec chmod 644 {} \; - - -# Now. IF we have docker, THEN build the docker image... - -if [ -x "$(which docker)" ]; then - - if [ -z "$DOCKER_FROM" ]; then - echo "" - read -p "Enter the docker image for the 'from' base: " DOCKER_FROM - echo "" - if [ -z "$DOCKER_FROM" ]; then - echo "DOCKER_FROM not set... so we can not build the docker image" - exit 1 - fi - fi - - if [ -z "$DOCKER_USERNAME" ]; then - echo "" - read -p "Enter a docker username: " DOCKER_USERNAME - echo "" - if [ -z "$DOCKER_USERNAME" ]; then - echo "DOCKER_USERNAME not set... so we can not build the docker image" - exit 1 - fi - fi - - export DOCKER_NAME="$DOCKER_USERNAME/pdf2htmlex:$PDF2HTMLEX_BRANCH-$BUILD_TIME" - - echo "export DOCKER_FROM=\"$DOCKER_FROM\"" >> ../../buildScripts/reSourceVersionEnvs - echo "export DOCKER_USERNAME=\"$DOCKER_USERNAME\"" >> ../../buildScripts/reSourceVersionEnvs - echo "export DOCKER_NAME=\"$DOCKER_NAME\"" >> ../../buildScripts/reSourceVersionEnvs - - delete_blacklisted - - cat > Dockerfile <> ../../buildScripts/reSourceVersionEnvs +echo "export DOCKER_USERNAME=\"$DOCKER_USERNAME\"" >> ../../buildScripts/reSourceVersionEnvs +echo "export DOCKER_NAME=\"$DOCKER_NAME\"" >> ../../buildScripts/reSourceVersionEnvs + +delete_blacklisted + +cat > Dockerfile < Date: Fri, 12 Jun 2020 14:53:44 +0100 Subject: [PATCH 30/69] corrected createImages --- buildScripts/createImages | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/buildScripts/createImages b/buildScripts/createImages index e07911f..ca0733e 100755 --- a/buildScripts/createImages +++ b/buildScripts/createImages @@ -7,12 +7,17 @@ set -ev ################# # do the creation -./buildScripts/reportEnvs || { echo 'reportEnvs FAILED' ; exit 1 ; } +./buildScripts/reportEnvs -./buildScripts/createAppImage || { echo 'createAppImage FAILED' ; exit 1 ; } +./buildScripts/createAppImage -./buildScripts/createDockerImage || { echo 'createDockerImage FAILED' ; exit 1 ; } +./buildScripts/createDebianPackage + +./buildScripts/createDockerImageInside + +if [ -x "$(which docker)" ]; then + ./buildScripts/createDockerImageOutside +fi -./buildScripts/createDebianPackage || { echo 'createDebianPackage FAILED' ; exit 1 ; } From f48822d81634434814713d10bd88fbfa91359ea6 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sat, 13 Jun 2020 19:38:51 +0100 Subject: [PATCH 31/69] corrected printing of error messages in test.py.in --- pdf2htmlEX/test/test.py.in | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pdf2htmlEX/test/test.py.in b/pdf2htmlEX/test/test.py.in index 16d619e..63f9983 100755 --- a/pdf2htmlEX/test/test.py.in +++ b/pdf2htmlEX/test/test.py.in @@ -13,8 +13,8 @@ class Common(object): """ PDF2HTMLEX_PATH = "@PDF2HTMLEX_PATH@" # defined in CMakeLists.txt if not os.path.isfile(PDF2HTMLEX_PATH) or not os.access(PDF2HTMLEX_PATH, os.X_OK): - print >> sys.stderr, "Cannot locate pdf2htmlEX executable, expected at ", PDF2HTMLEX_PATH, - ". Make sure source was built before running this test." + print( sys.stderr, "Cannot locate pdf2htmlEX executable, expected at ", PDF2HTMLEX_PATH, + ". Make sure source was built before running this test." ) exit(1) SRC_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) From 4171aef19e73469420df02ada20f87843293a804 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sun, 14 Jun 2020 10:35:20 +0100 Subject: [PATCH 32/69] added explicit install of wget in installAutomaticTests --- pdf2htmlEX/test/installAutomaticTestSoftware | 1 + 1 file changed, 1 insertion(+) diff --git a/pdf2htmlEX/test/installAutomaticTestSoftware b/pdf2htmlEX/test/installAutomaticTestSoftware index 032c2d0..2b53ad2 100755 --- a/pdf2htmlEX/test/installAutomaticTestSoftware +++ b/pdf2htmlEX/test/installAutomaticTestSoftware @@ -10,6 +10,7 @@ export DEBIAN_FRONTEND=noninteractive # Start by making sure all required apt packages exist # sudo apt -y install \ + wget \ diffutils \ zip \ python3 \ From 2853b9e36bca42cc64a562ef6ee0bfac43c4c73e Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sun, 14 Jun 2020 16:46:05 +0100 Subject: [PATCH 33/69] refactored existing python3 test_output.py tests to use only bash and pdf2htmlEX so we do not need to load extra fonts by accident --- pdf2htmlEX/test/runLocalTests | 2 +- pdf2htmlEX/test/runLocalTestsPython | 7 + pdf2htmlEX/test/testOutput | 202 ++++++++++++++++++++++++++++ 3 files changed, 210 insertions(+), 1 deletion(-) create mode 100755 pdf2htmlEX/test/runLocalTestsPython create mode 100755 pdf2htmlEX/test/testOutput diff --git a/pdf2htmlEX/test/runLocalTests b/pdf2htmlEX/test/runLocalTests index f8db9c7..f3bf575 100755 --- a/pdf2htmlEX/test/runLocalTests +++ b/pdf2htmlEX/test/runLocalTests @@ -4,4 +4,4 @@ set -ev # This bash script runs the (simple non-browser) tests -python3 test_output.py +./testOutput diff --git a/pdf2htmlEX/test/runLocalTestsPython b/pdf2htmlEX/test/runLocalTestsPython new file mode 100755 index 0000000..f8db9c7 --- /dev/null +++ b/pdf2htmlEX/test/runLocalTestsPython @@ -0,0 +1,7 @@ +#!/bin/bash + +set -ev + +# This bash script runs the (simple non-browser) tests + +python3 test_output.py diff --git a/pdf2htmlEX/test/testOutput b/pdf2htmlEX/test/testOutput new file mode 100755 index 0000000..a74afd1 --- /dev/null +++ b/pdf2htmlEX/test/testOutput @@ -0,0 +1,202 @@ +#!/bin/bash + +# This bash script checks that pdf2htmlEX does not crash, and produces +# correct files. Do not check the content of the files. + +# We use a bash script to implement the python3 test_output.py so that we +# do not need to install any extra packages to test this functionality. + +# This is how we run pdf2htmlEX on a particular file, and arguments. +# +function runPdf2htmlEX { + # collect the arguments + # + export LAST_TEST_NAME="$*" + # + pdfFileName=$1 + # + shift + arguments=$* + + # clear out the TMPDIR + # + rm -rf $PDF2HTMLEX_TMPDIR + mkdir -p $PDF2HTMLEX_TMPDIR + # + # now run pdf2htmlEX to produce the output files + # + echo "test: [$LAST_TEST_NAME]" + echo "---" + $PDF2HTMLEX_PATH \ + --data-dir $PDF2HTMLEX_DATDIR \ + --dest-dir $PDF2HTMLEX_TMPDIR \ + $PDF2HTMLEX_TEST_DIR/test_output/$pdfFileName \ + $arguments +} + +# This is how we test for expected output files +# +function hasExpectedFiles { + filesFound="true" + for anExpectedFile in $1 ; do + if ! test -r $PDF2HTMLEX_TMPDIR/$anExpectedFile ; then + echo "NOT FOUND [$PDF2HTMLEX_TMPDIR/$anExpectedFile]" + filesFound="false" + fi + done + if test $filesFound = "true" ; then + echo "SUCCESS: $LAST_TEST_NAME" >> testOutputResults + else + echo "FAILURE: $LAST_TEST_NAME" >> testOutputResults + fi + echo "---" + echo "" +} + +# This is how we copy a file omitting lines between '#TEST_IGNORE_BEGIN' +# and '#TEST_IGNORE_END' (we pipe the file in via stdin and save it via +# stdout) +# +function copy_TEST_IGNORE_file { + skipLine=echo + while IFS= read -r line ; do + if echo $line | grep -q "TEST_IGNORE_BEGIN" ; then skipLine=true ; fi + if echo $line | grep -q "TEST_IGNORE_END" ; then skipLine=echo ; fi + $skipLine "$line" + done +} + +if test -z "$PDF2HTMLEX_PATH" ; then + echo "PANIC: we do not know where to find the pdf2htmlEX executable" + exit 1 +fi + +if test -z "$PDF2HTMLEX_DATDIR" ; then + export PDF2HTMLEX_DATDIR=/tmp/pdf2htmlex/dat +fi + +mkdir -p $PDF2HTMLEX_DATDIR + +if test -z "$PDF2HTMLEX_TMPDIR" ; then + export PDF2HTMLEX_TMPDIR=/tmp/pdf2htmlex/tmp +fi + +mkdir -p $PDF2HTMLEX_TMPDIR + +if test -z "$PDF2HTMLEX_TEST_DIR" ; then + export PDF2HTMLEX_TEST_DIR=. +fi + +mkdir -p $PDF2HTMLEX_TEST_DIR + +# Make sure any previous testOutputResults are cleared +# +rm -f testOutputResults + +# setup the correct data files +# +cat $PDF2HTMLEX_TEST_DIR/../share/manifest | \ + copy_TEST_IGNORE_file > $PDF2HTMLEX_DATDIR/manifest + +cp $PDF2HTMLEX_TEST_DIR/../share/base.min.css $PDF2HTMLEX_DATDIR +cp $PDF2HTMLEX_TEST_DIR/../share/fancy.min.css $PDF2HTMLEX_DATDIR + +# Do the tests +# +echo "" +echo "-------------------------------------------------------" +echo "running testOutput tests (simple non-browser tests)" +echo "-------------------------------------------------------" +echo "" + +export LAST_TEST_NAME="unknown" +# +runPdf2htmlEX '1-page.pdf' --version + +runPdf2htmlEX '1-page.pdf' +hasExpectedFiles '1-page.html' + +runPdf2htmlEX '2-pages.pdf' +hasExpectedFiles '2-pages.html' + +runPdf2htmlEX '1-page.pdf' 'foo.html' +hasExpectedFiles 'foo.html' + +runPdf2htmlEX '2-pages.pdf' 'foo.html' +hasExpectedFiles 'foo.html' + +runPdf2htmlEX '1-page.pdf' '--split-pages=1' +hasExpectedFiles '1-page.html 1-page1.page' + +runPdf2htmlEX '3-pages.pdf' '--split-pages=1' +hasExpectedFiles '3-pages.html 3-pages1.page 3-pages2.page 3-pages3.page' + +runPdf2htmlEX '1-page.pdf' '--split-pages=1 --page-filename=foo.xyz' +hasExpectedFiles '1-page.html foo1.xyz' + +runPdf2htmlEX '3-pages.pdf' '--split-pages=1 --page-filename=foo.xyz' +hasExpectedFiles '3-pages.html foo1.xyz foo2.xyz foo3.xyz' + +runPdf2htmlEX '3-pages.pdf' '--split-pages=1 --page-filename=fo%do.xyz' +hasExpectedFiles '3-pages.html fo1o.xyz fo2o.xyz fo3o.xyz' + +runPdf2htmlEX '3-pages.pdf' '--split-pages=1 --page-filename=fo%03do.xyz' +hasExpectedFiles '3-pages.html fo001o.xyz fo002o.xyz fo003o.xyz' + +runPdf2htmlEX '3-pages.pdf' '--split-pages=1 --page-filename=f%do%do.xyz' +hasExpectedFiles '3-pages.html f1o%do.xyz f2o%do.xyz f3o%do.xyz' + +runPdf2htmlEX '3-pages.pdf' '--split-pages=1 --page-filename=f%soo.xyz' +hasExpectedFiles '3-pages.html f%soo1.xyz f%soo2.xyz f%soo3.xyz' + +runPdf2htmlEX '3-pages.pdf' '--split-pages=1 --page-filename=f%poo.xyz' +hasExpectedFiles '3-pages.html f%poo1.xyz f%poo2.xyz f%poo3.xyz' + +runPdf2htmlEX '3-pages.pdf' '--split-pages=1 --page-filename=f%noo.xyz' +hasExpectedFiles '3-pages.html f%noo1.xyz f%noo2.xyz f%noo3.xyz' + +runPdf2htmlEX '3-pages.pdf' '--split-pages=1 --page-filename=f%%oo.xyz' +hasExpectedFiles '3-pages.html f%%oo1.xyz f%%oo2.xyz f%%oo3.xyz' + +runPdf2htmlEX '3-pages.pdf' '--split-pages=1 --page-filename=f%%o%do.xyz' +hasExpectedFiles '3-pages.html f%%o1o.xyz f%%o2o.xyz f%%o3o.xyz' + +runPdf2htmlEX '3-pages.pdf' '--split-pages=1 --page-filename=fo%do%%.xyz' +hasExpectedFiles '3-pages.html fo1o%%.xyz fo2o%%.xyz fo3o%%.xyz' + +runPdf2htmlEX '3-pages.pdf' '--split-pages=1 --page-filename=f%02%doo.xyz' +hasExpectedFiles '3-pages.html f%021oo.xyz f%022oo.xyz f%023oo.xyz' + +runPdf2htmlEX '1-page.pdf' '--split-pages=1 --page-filename=foo' +hasExpectedFiles '1-page.html foo1' + +runPdf2htmlEX '2-pages.pdf' 'foo%d.html' +hasExpectedFiles 'foo%d.html' + +runPdf2htmlEX '2-pages.pdf' 'foo%p.html' +hasExpectedFiles 'foo%p.html' + +runPdf2htmlEX '2-pages.pdf' 'foo%n.html' +hasExpectedFiles 'foo%n.html' + +runPdf2htmlEX '2-pages.pdf' 'foo%%.html' +hasExpectedFiles 'foo%%.html' + +runPdf2htmlEX 'issue501' '--split-pages=1 --embed-css=0' +hasExpectedFiles '' + +# Let the user know what the testOutputResults were +# +echo "" +echo "-------------------------------------------------------" +echo "testOutput results:" +echo "-------------------------------------------------------" +cat testOutputResults +echo "-------------------------------------------------------" +echo "" + +# fail the script IF we find the work 'FAILURE' in the testOutputResults +# +if grep FAILURE testOutputResults ; then + exit 1 +fi From df62ee7f214ab0552fc6e202a9ca22d26d5a008b Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sun, 14 Jun 2020 17:07:34 +0100 Subject: [PATCH 34/69] renamed runLocalTests --- pdf2htmlEX/test/{runLocalTests => runLocalTestsBash} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename pdf2htmlEX/test/{runLocalTests => runLocalTestsBash} (100%) diff --git a/pdf2htmlEX/test/runLocalTests b/pdf2htmlEX/test/runLocalTestsBash similarity index 100% rename from pdf2htmlEX/test/runLocalTests rename to pdf2htmlEX/test/runLocalTestsBash From 16c88c49c482aed7295e87ca20e67547a7e5944e Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Mon, 15 Jun 2020 14:27:28 +0100 Subject: [PATCH 35/69] browser_tests now use pre computed html --- pdf2htmlEX/test/browser_tests.py | 20 ++++-- pdf2htmlEX/test/produceHtmlForBrowserTests | 82 ++++++++++++++++++++++ pdf2htmlEX/test/test.py.in | 1 + 3 files changed, 98 insertions(+), 5 deletions(-) create mode 100755 pdf2htmlEX/test/produceHtmlForBrowserTests diff --git a/pdf2htmlEX/test/browser_tests.py b/pdf2htmlEX/test/browser_tests.py index 93906ad..b839692 100644 --- a/pdf2htmlEX/test/browser_tests.py +++ b/pdf2htmlEX/test/browser_tests.py @@ -37,13 +37,23 @@ class BrowserTests(Common): ref_htmlfolder = os.path.join(self.TEST_DATA_DIR, basefilename) ref_htmlfilename = os.path.join(ref_htmlfolder, htmlfilename) out_htmlfilename = os.path.join(self.OUTDIR, htmlfilename) - - pdf2htmlEX_args = self.DEFAULT_PDF2HTMLEX_ARGS + args + [ + pre_htmlfilename = os.path.join(self.PREDIR, htmlfilename) + + try: + # see if we have pre-compiled the html file... + # if so simply copy it into place + # + shutil.copy(pre_htmlfilename, out_htmlfilename) + except: + # we have not pre-compiled the html file + # so create it using pdf2htmlEX + # + pdf2htmlEX_args = self.DEFAULT_PDF2HTMLEX_ARGS + args + [ os.path.join(self.TEST_DATA_DIR, filename), htmlfilename ] - result = self.run_pdf2htmlEX(pdf2htmlEX_args) - - self.assertIn(htmlfilename, result['output_files'], 'HTML file is not generated') + result = self.run_pdf2htmlEX(pdf2htmlEX_args) + # + self.assertIn(htmlfilename, result['output_files'], 'HTML file is not generated') if self.GENERATING_MODE: # copy generated html files diff --git a/pdf2htmlEX/test/produceHtmlForBrowserTests b/pdf2htmlEX/test/produceHtmlForBrowserTests new file mode 100755 index 0000000..3ef8f0f --- /dev/null +++ b/pdf2htmlEX/test/produceHtmlForBrowserTests @@ -0,0 +1,82 @@ +#!/bin/bash + +# This bash script walks through the browser_tests directory running +# pdf2htmlEX on each *.pdf file. + +# This is how we run pdf2htmlEX on a particular file, and arguments. +# +function runPdf2htmlEX { + pdfFileName=$1 + htmlFileName=$(echo $pdfFileName | cut -d'.' -f1).html + arguments=$2 + # + echo "" + echo "---" + echo " pdfFileName: [$pdfFileName]" + echo "htmlFileName: [$htmlFileName]" + echo " arguments: [$arguments]" + # + # now run pdf2htmlEX to produce the output files + # + echo $PDF2HTMLEX_PATH \ + --data-dir=$PDF2HTMLEX_DATDIR \ + --dest-dir $PDF2HTMLEX_TMPDIR \ + --fit-width=800 --last-page=1 \ + $arguments \ + browser_tests/$pdfFileName \ + $htmlFileName + # + $PDF2HTMLEX_PATH \ + --data-dir=$PDF2HTMLEX_DATDIR \ + --dest-dir $PDF2HTMLEX_TMPDIR \ + --fit-width=800 --last-page=1 \ + $arguments \ + browser_tests/$pdfFileName \ + $htmlFileName +} + +if test -z "$PDF2HTMLEX_PATH" ; then + echo "PANIC: we do not know where to find the pdf2htmlEX executable" + exit 1 +fi + +if test -z "$PDF2HTMLEX_DATDIR" ; then + export PDF2HTMLEX_DATDIR=/tmp/pdf2htmlEX/dat +fi + +if test -z "$PDF2HTMLEX_TMPDIR" ; then + export PDF2HTMLEX_TMPDIR=/tmp/pdf2htmlEX/tmp +fi + +if test -z "$PDF2HTMLEX_PREDIR" ; then + export PDF2HTMLEX_PREDIR=/tmp/pdf2htmlEX/pre +fi + +# clear out the TMPDIR +# +rm -rf $PDF2HTMLEX_TMPDIR +mkdir -p $PDF2HTMLEX_TMPDIR +# + +runPdf2htmlEX 'test_fail.pdf' + +runPdf2htmlEX 'basic_text.pdf' + +runPdf2htmlEX 'geneve_1564.pdf' + +runPdf2htmlEX 'text_visibility.pdf' '--correct-text-visibility=1' + +runPdf2htmlEX 'with_form.pdf' '--process-form=1' + +runPdf2htmlEX 'invalid_unicode_issue477.pdf' + +runPdf2htmlEX 'svg_background_with_page_rotation_issue402.pdf' '--bg-format=svg' + +runPdf2htmlEX 'fontfile3_opentype.pdf' + +# clear out the PREDIR +# +rm -rf $PDF2HTMLEX_PREDIR +mkdir -p $PDF2HTMLEX_PREDIR +# +cp $PDF2HTMLEX_TMPDIR/* $PDF2HTMLEX_PREDIR diff --git a/pdf2htmlEX/test/test.py.in b/pdf2htmlEX/test/test.py.in index 63f9983..8748d64 100755 --- a/pdf2htmlEX/test/test.py.in +++ b/pdf2htmlEX/test/test.py.in @@ -28,6 +28,7 @@ class Common(object): PNGDIR = "@PDF2HTMLEX_PNGDIR@" DATDIR = "@PDF2HTMLEX_DATDIR@" OUTDIR = "@PDF2HTMLEX_OUTDIR@" + PREDIR = "@PDF2HTMLEX_PREDIR@" HTMDIR = "@PDF2HTMLEX_HTMDIR@" def setUp(self): From 7f2bb46407e0d59e3b21118ea229c645478951a1 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Tue, 16 Jun 2020 15:48:01 +0100 Subject: [PATCH 36/69] explitictly split PDF2HTMLEX_PATH in test.py.in to allow for arguments in path --- pdf2htmlEX/test/test.py.in | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/pdf2htmlEX/test/test.py.in b/pdf2htmlEX/test/test.py.in index 8748d64..5eab842 100755 --- a/pdf2htmlEX/test/test.py.in +++ b/pdf2htmlEX/test/test.py.in @@ -63,10 +63,14 @@ class Common(object): shutil.rmtree(self.TMPDIR, ignore_errors=False, onerror=None) os.mkdir(self.TMPDIR) - args = [Common.PDF2HTMLEX_PATH, - '--data-dir', self.DATDIR, + # To allow the user to specify additional command line arguments + # in the path, we split the PATH on whitespace and then append our + # arguments. + # + args = Common.PDF2HTMLEX_PATH.split() + + [ '--data-dir', self.DATDIR, '--dest-dir', self.TMPDIR - ] + args + ] + args with open(os.devnull, 'w') as fnull: return_code = subprocess.call(list(map(str, args)), stderr=fnull) From 7d2c5df9d73d04005d76c221a448dc7964ddfdc7 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Tue, 16 Jun 2020 16:26:19 +0100 Subject: [PATCH 37/69] corrected runTests to use pdf2htmlEX/test/runLocalTestsBash --- buildScripts/runTests | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/buildScripts/runTests b/buildScripts/runTests index a218cca..1c7da9a 100755 --- a/buildScripts/runTests +++ b/buildScripts/runTests @@ -21,7 +21,7 @@ cd pdf2htmlEX/test # ./installAutomaticTestSoftware -./runLocalTests +./runLocalTestsBash ./runLocalBrowserTests From 402273ff2a22dc0a08b1bb9a1d92625d6dd233f4 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Tue, 16 Jun 2020 16:38:40 +0100 Subject: [PATCH 38/69] second attempt to fix test.py.in --- pdf2htmlEX/test/test.py.in | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/pdf2htmlEX/test/test.py.in b/pdf2htmlEX/test/test.py.in index 5eab842..e0aaa20 100755 --- a/pdf2htmlEX/test/test.py.in +++ b/pdf2htmlEX/test/test.py.in @@ -67,8 +67,7 @@ class Common(object): # in the path, we split the PATH on whitespace and then append our # arguments. # - args = Common.PDF2HTMLEX_PATH.split() + - [ '--data-dir', self.DATDIR, + args = Common.PDF2HTMLEX_PATH.split() + [ '--data-dir', self.DATDIR, '--dest-dir', self.TMPDIR ] + args From eb63aa06a45bc68e15eab26c46ecdaf21f82b9d2 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Wed, 17 Jun 2020 11:32:06 +0100 Subject: [PATCH 39/69] corrected setup of data directory for tests --- pdf2htmlEX/test/produceHtmlForBrowserTests | 25 ++++++++++++++++++++++ pdf2htmlEX/test/testOutput | 12 +++++++---- 2 files changed, 33 insertions(+), 4 deletions(-) diff --git a/pdf2htmlEX/test/produceHtmlForBrowserTests b/pdf2htmlEX/test/produceHtmlForBrowserTests index 3ef8f0f..2fdce82 100755 --- a/pdf2htmlEX/test/produceHtmlForBrowserTests +++ b/pdf2htmlEX/test/produceHtmlForBrowserTests @@ -35,6 +35,23 @@ function runPdf2htmlEX { $htmlFileName } +# This is how we copy a file omitting lines between '#TEST_IGNORE_BEGIN' +# and '#TEST_IGNORE_END' (we pipe the file in via stdin and save it via +# stdout) +# +function copy_TEST_IGNORE_file { + skipLine=echo + while IFS= read -r line ; do + if echo $line | grep -q "^#TEST_IGNORE_BEGIN" ; then + skipLine=true + elif echo $line | grep -q "^#TEST_IGNORE_END" ; then + skipLine=echo + else + $skipLine "$line" + fi + done +} + if test -z "$PDF2HTMLEX_PATH" ; then echo "PANIC: we do not know where to find the pdf2htmlEX executable" exit 1 @@ -58,6 +75,14 @@ rm -rf $PDF2HTMLEX_TMPDIR mkdir -p $PDF2HTMLEX_TMPDIR # +# setup the correct data files +# +cat $PDF2HTMLEX_TEST_DIR/../share/manifest | \ + copy_TEST_IGNORE_file > $PDF2HTMLEX_DATDIR/manifest + +cp $PDF2HTMLEX_TEST_DIR/../share/base.min.css $PDF2HTMLEX_DATDIR +cp $PDF2HTMLEX_TEST_DIR/../test/fancy.min.css $PDF2HTMLEX_DATDIR + runPdf2htmlEX 'test_fail.pdf' runPdf2htmlEX 'basic_text.pdf' diff --git a/pdf2htmlEX/test/testOutput b/pdf2htmlEX/test/testOutput index a74afd1..c18ebdb 100755 --- a/pdf2htmlEX/test/testOutput +++ b/pdf2htmlEX/test/testOutput @@ -60,9 +60,13 @@ function hasExpectedFiles { function copy_TEST_IGNORE_file { skipLine=echo while IFS= read -r line ; do - if echo $line | grep -q "TEST_IGNORE_BEGIN" ; then skipLine=true ; fi - if echo $line | grep -q "TEST_IGNORE_END" ; then skipLine=echo ; fi - $skipLine "$line" + if echo $line | grep -q "^#TEST_IGNORE_BEGIN" ; then + skipLine=true + elif echo $line | grep -q "^#TEST_IGNORE_END" ; then + skipLine=echo + else + $skipLine "$line" + fi done } @@ -99,7 +103,7 @@ cat $PDF2HTMLEX_TEST_DIR/../share/manifest | \ copy_TEST_IGNORE_file > $PDF2HTMLEX_DATDIR/manifest cp $PDF2HTMLEX_TEST_DIR/../share/base.min.css $PDF2HTMLEX_DATDIR -cp $PDF2HTMLEX_TEST_DIR/../share/fancy.min.css $PDF2HTMLEX_DATDIR +cp $PDF2HTMLEX_TEST_DIR/../test/fancy.min.css $PDF2HTMLEX_DATDIR # Do the tests # From 27df8a4262dde354ac61abe26969a72c276003cb Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Wed, 17 Jun 2020 14:14:55 +0100 Subject: [PATCH 40/69] corrected produceHtmlForBrowserTests --- pdf2htmlEX/test/produceHtmlForBrowserTests | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/pdf2htmlEX/test/produceHtmlForBrowserTests b/pdf2htmlEX/test/produceHtmlForBrowserTests index 2fdce82..6225aab 100755 --- a/pdf2htmlEX/test/produceHtmlForBrowserTests +++ b/pdf2htmlEX/test/produceHtmlForBrowserTests @@ -69,6 +69,10 @@ if test -z "$PDF2HTMLEX_PREDIR" ; then export PDF2HTMLEX_PREDIR=/tmp/pdf2htmlEX/pre fi +if test -z "$PDF2HTMLEX_TEST_DIR" ; then + export PDF2HTMLEX_TEST_DIR=. +fi + # clear out the TMPDIR # rm -rf $PDF2HTMLEX_TMPDIR From cb00d5a5d24b2178164ca9f4af67da3d7ecfd3a6 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Wed, 17 Jun 2020 16:14:24 +0100 Subject: [PATCH 41/69] attempt to debug appImage problems --- pdf2htmlEX/test/test.py.in | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/pdf2htmlEX/test/test.py.in b/pdf2htmlEX/test/test.py.in index e0aaa20..8f5f7dc 100755 --- a/pdf2htmlEX/test/test.py.in +++ b/pdf2htmlEX/test/test.py.in @@ -70,7 +70,8 @@ class Common(object): args = Common.PDF2HTMLEX_PATH.split() + [ '--data-dir', self.DATDIR, '--dest-dir', self.TMPDIR ] + args - + print("run_pdf2htmlEX args: [", args, "]") + with open(os.devnull, 'w') as fnull: return_code = subprocess.call(list(map(str, args)), stderr=fnull) From a5dd4865aef6e3d73f98b70484df05176006c0bc Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Wed, 17 Jun 2020 19:56:35 +0100 Subject: [PATCH 42/69] corrected test.py.in from failing with appImages complex PATH and ARGS --- pdf2htmlEX/test/test.py.in | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/pdf2htmlEX/test/test.py.in b/pdf2htmlEX/test/test.py.in index 8f5f7dc..163626f 100755 --- a/pdf2htmlEX/test/test.py.in +++ b/pdf2htmlEX/test/test.py.in @@ -12,8 +12,10 @@ class Common(object): Variables and methods for common use in different tests. """ PDF2HTMLEX_PATH = "@PDF2HTMLEX_PATH@" # defined in CMakeLists.txt - if not os.path.isfile(PDF2HTMLEX_PATH) or not os.access(PDF2HTMLEX_PATH, os.X_OK): - print( sys.stderr, "Cannot locate pdf2htmlEX executable, expected at ", PDF2HTMLEX_PATH, + PDF2HTMLEX_EXE = PDF2HTMLEX_PATH.split() + PDF2HTMLEX_EXE = PDF2HTMLEX_EXE[0] + if not os.path.isfile(PDF2HTMLEX_EXE) or not os.access(PDF2HTMLEX_EXE, os.X_OK): + print( sys.stderr, "Cannot locate pdf2htmlEX executable, expected at ", PDF2HTMLEX_EXE, ". Make sure source was built before running this test." ) exit(1) @@ -70,7 +72,6 @@ class Common(object): args = Common.PDF2HTMLEX_PATH.split() + [ '--data-dir', self.DATDIR, '--dest-dir', self.TMPDIR ] + args - print("run_pdf2htmlEX args: [", args, "]") with open(os.devnull, 'w') as fnull: return_code = subprocess.call(list(map(str, args)), stderr=fnull) From d3550d5ea85a1c6d07f67f45d3aadb6ba0972e38 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Fri, 19 Jun 2020 09:34:51 +0100 Subject: [PATCH 43/69] first attempt to fix docker outside ldconfig problem and correct travis script --- buildScripts/createDockerImageOutside | 2 +- buildScripts/travisLinuxDoItAll | 3 +++ 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/buildScripts/createDockerImageOutside b/buildScripts/createDockerImageOutside index 40cad36..390062c 100755 --- a/buildScripts/createDockerImageOutside +++ b/buildScripts/createDockerImageOutside @@ -53,7 +53,7 @@ delete_blacklisted cat > Dockerfile < Date: Fri, 19 Jun 2020 10:24:16 +0100 Subject: [PATCH 44/69] fix travis 2 --- buildScripts/travisLinuxDoItAll | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/buildScripts/travisLinuxDoItAll b/buildScripts/travisLinuxDoItAll index a3da7c4..e5f6784 100755 --- a/buildScripts/travisLinuxDoItAll +++ b/buildScripts/travisLinuxDoItAll @@ -34,6 +34,7 @@ export DEBIAN_FRONTEND=noninteractive ./buildScripts/runTests ./buildScripts/createAppImage ./buildScripts/createDebianPackage -./buildScripts/createDockerImage +./buildScripts/createDockerImageInside +./buildScripts/createDockerImageOutside ./buildScripts/uploadGitHubRelease ./buildScripts/uploadDockerImage From 8f58db5f7f1d3cff5e25b92310b8aee74a112358 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Fri, 19 Jun 2020 11:19:43 +0100 Subject: [PATCH 45/69] travis fix 3 --- buildScripts/createDebianPackage | 2 +- buildScripts/versionEnvs | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/buildScripts/createDebianPackage b/buildScripts/createDebianPackage index 6ddb7b1..499d7aa 100755 --- a/buildScripts/createDebianPackage +++ b/buildScripts/createDebianPackage @@ -67,7 +67,7 @@ find $DEBDIR -type f | xargs md5sum > $md5sumsFile # Accumulate the control file information # -versionValue=$(git describe --abbrev=0) +versionValue=$PDF2HTMLEX_VERSION-$PDF2HTMLEX_BRANCH-$BUILD_DIST-$BUILD_TIME architectureValue=$(dpkg-architecture -q DEB_BUILD_ARCH_CPU) maintainerValue="$(git config --get user.name) <$(git config --get user.email)>" diff --git a/buildScripts/versionEnvs b/buildScripts/versionEnvs index 0bc662a..26e0366 100755 --- a/buildScripts/versionEnvs +++ b/buildScripts/versionEnvs @@ -6,6 +6,8 @@ # see: https://poppler.freedesktop.org/releases.html # current working: 0.83.0 +export PDF2HTMLEX_VERSION=v0.18.8-alpha + export POPPLER_VERSION=poppler-0.89.0 #export POPPLER_VERSION=poppler-0.88.0 #export POPPLER_VERSION=poppler-0.87.0 From 2b51b0e62d00f0ef96a0b3837150bc9241a32e28 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Fri, 19 Jun 2020 11:32:45 +0100 Subject: [PATCH 46/69] attempt to build docker image from deb archive --- buildScripts/createDockerUbuntuImageFromDeb | 66 +++++++++++++++++++++ 1 file changed, 66 insertions(+) create mode 100755 buildScripts/createDockerUbuntuImageFromDeb diff --git a/buildScripts/createDockerUbuntuImageFromDeb b/buildScripts/createDockerUbuntuImageFromDeb new file mode 100755 index 0000000..adc3a87 --- /dev/null +++ b/buildScripts/createDockerUbuntuImageFromDeb @@ -0,0 +1,66 @@ +#!/bin/bash + +# This bash script creates a docker image from an existing pdf2htmlEX +# +# This is the part which must be run *outside* of any docker container. + +echo "" +echo "-------------------------------------------------------------------" +echo "CREATING pdf2htmlEX Docker Image (from deb archive)" +echo "-------------------------------------------------------------------" +echo "" + +# Collect everything that will be needed... + +source buildScripts/reSourceVersionEnvs + +source buildScripts/dockerFunctions + +set -ev + +mkdir -p imageBuild/dockerDir + +cd imageBuild/dockerDir + +cp $PWORKSPACE/$DIST/$DPKG_NAME . + +if [ -z "$DOCKER_FROM" ]; then + echo "" + read -p "Enter the docker image for the 'from' base: " DOCKER_FROM + echo "" + if [ -z "$DOCKER_FROM" ]; then + echo "DOCKER_FROM not set... so we can not build the docker image" + exit 1 + fi +fi + +if [ -z "$DOCKER_USERNAME" ]; then + echo "" + read -p "Enter a docker username: " DOCKER_USERNAME + echo "" + if [ -z "$DOCKER_USERNAME" ]; then + echo "DOCKER_USERNAME not set... so we can not build the docker image" + exit 1 + fi +fi + +export DOCKER_NAME="$DOCKER_USERNAME/pdf2htmlex:$PDF2HTMLEX_BRANCH-$BUILD_TIME" + +echo "export DOCKER_FROM=\"$DOCKER_FROM\"" >> ../../buildScripts/reSourceVersionEnvs +echo "export DOCKER_USERNAME=\"$DOCKER_USERNAME\"" >> ../../buildScripts/reSourceVersionEnvs +echo "export DOCKER_NAME=\"$DOCKER_NAME\"" >> ../../buildScripts/reSourceVersionEnvs + +cat > Dockerfile < Date: Fri, 19 Jun 2020 11:37:25 +0100 Subject: [PATCH 47/69] corrected PDF2HTMLEX_VERSION --- buildScripts/versionEnvs | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/buildScripts/versionEnvs b/buildScripts/versionEnvs index 26e0366..f5ad0fa 100755 --- a/buildScripts/versionEnvs +++ b/buildScripts/versionEnvs @@ -52,10 +52,11 @@ export BUILD_TIME="$(date +%Y_%m_%d-%H_%M_%S)" export MACHINE_ARCH="$(uname -m)" -echo "export POPPLER_VERSION=\"$POPPLER_VERSION\"" > buildScripts/reSourceVersionEnvs -echo "export FONTFORGE_VERSION=\"$FONTFORGE_VERSION\"" >> buildScripts/reSourceVersionEnvs -echo "export PDF2HTMLEX_BRANCH=\"$PDF2HTMLEX_BRANCH\"" >> buildScripts/reSourceVersionEnvs -echo "export MACHINE_ARCH=\"$MACHINE_ARCH\"" >> buildScripts/reSourceVersionEnvs -echo "export BUILD_DIST=\"$BUILD_DIST\"" >> buildScripts/reSourceVersionEnvs -echo "export BUILD_TIME=\"$BUILD_TIME\"" >> buildScripts/reSourceVersionEnvs -echo "export PDF2HTMLEX_PREFIX=\"$PDF2HTMLEX_PREFIX\"" >> buildScripts/reSourceVersionEnvs +echo "export PDF2HTMLEX_VERSION=\"$PDF2HTMLEX_VERSION\"" > buildScripts/reSourceVersionEnvs +echo "export POPPLER_VERSION=\"$POPPLER_VERSION\"" >> buildScripts/reSourceVersionEnvs +echo "export FONTFORGE_VERSION=\"$FONTFORGE_VERSION\"" >> buildScripts/reSourceVersionEnvs +echo "export PDF2HTMLEX_BRANCH=\"$PDF2HTMLEX_BRANCH\"" >> buildScripts/reSourceVersionEnvs +echo "export MACHINE_ARCH=\"$MACHINE_ARCH\"" >> buildScripts/reSourceVersionEnvs +echo "export BUILD_DIST=\"$BUILD_DIST\"" >> buildScripts/reSourceVersionEnvs +echo "export BUILD_TIME=\"$BUILD_TIME\"" >> buildScripts/reSourceVersionEnvs +echo "export PDF2HTMLEX_PREFIX=\"$PDF2HTMLEX_PREFIX\"" >> buildScripts/reSourceVersionEnvs From a9d178ea0be2b690ce3c50d30dfb812d0e00f1ad Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Fri, 19 Jun 2020 12:17:14 +0100 Subject: [PATCH 48/69] another attempt to solve dpkg-deb invalid characters in version problem --- buildScripts/createDebianPackage | 2 +- buildScripts/versionEnvs | 5 ++++- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/buildScripts/createDebianPackage b/buildScripts/createDebianPackage index 499d7aa..1dbff39 100755 --- a/buildScripts/createDebianPackage +++ b/buildScripts/createDebianPackage @@ -67,7 +67,7 @@ find $DEBDIR -type f | xargs md5sum > $md5sumsFile # Accumulate the control file information # -versionValue=$PDF2HTMLEX_VERSION-$PDF2HTMLEX_BRANCH-$BUILD_DIST-$BUILD_TIME +versionValue=$PDF2HTMLEX_VERSION.$PDF2HTMLEX_BRANCH.$BUILD_DIST.$BUILD_DATE architectureValue=$(dpkg-architecture -q DEB_BUILD_ARCH_CPU) maintainerValue="$(git config --get user.name) <$(git config --get user.email)>" diff --git a/buildScripts/versionEnvs b/buildScripts/versionEnvs index f5ad0fa..d8c1da6 100755 --- a/buildScripts/versionEnvs +++ b/buildScripts/versionEnvs @@ -6,7 +6,7 @@ # see: https://poppler.freedesktop.org/releases.html # current working: 0.83.0 -export PDF2HTMLEX_VERSION=v0.18.8-alpha +export PDF2HTMLEX_VERSION=0.18.8.alpha export POPPLER_VERSION=poppler-0.89.0 #export POPPLER_VERSION=poppler-0.88.0 @@ -48,6 +48,8 @@ source /etc/lsb-release export BUILD_DIST=$DISTRIB_CODENAME +export BUILD_DATE="$(date +%Y%m%d)" + export BUILD_TIME="$(date +%Y_%m_%d-%H_%M_%S)" export MACHINE_ARCH="$(uname -m)" @@ -58,5 +60,6 @@ echo "export FONTFORGE_VERSION=\"$FONTFORGE_VERSION\"" >> buildScripts/reSourc echo "export PDF2HTMLEX_BRANCH=\"$PDF2HTMLEX_BRANCH\"" >> buildScripts/reSourceVersionEnvs echo "export MACHINE_ARCH=\"$MACHINE_ARCH\"" >> buildScripts/reSourceVersionEnvs echo "export BUILD_DIST=\"$BUILD_DIST\"" >> buildScripts/reSourceVersionEnvs +echo "export BUILD_DATE=\"$BUILD_DATE\"" >> buildScripts/reSourceVersionEnvs echo "export BUILD_TIME=\"$BUILD_TIME\"" >> buildScripts/reSourceVersionEnvs echo "export PDF2HTMLEX_PREFIX=\"$PDF2HTMLEX_PREFIX\"" >> buildScripts/reSourceVersionEnvs From b67134375728858b65ebb815c8442cbf1b87c5a6 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Fri, 19 Jun 2020 12:54:04 +0100 Subject: [PATCH 49/69] corrected docker build from debian archive --- buildScripts/createDockerUbuntuImageFromDeb | 8 +++++--- buildScripts/reportEnvs | 2 ++ 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/buildScripts/createDockerUbuntuImageFromDeb b/buildScripts/createDockerUbuntuImageFromDeb index adc3a87..2cee46a 100755 --- a/buildScripts/createDockerUbuntuImageFromDeb +++ b/buildScripts/createDockerUbuntuImageFromDeb @@ -18,11 +18,13 @@ source buildScripts/dockerFunctions set -ev -mkdir -p imageBuild/dockerDir +cd imageBuild -cd imageBuild/dockerDir +mkdir -p dockerDir -cp $PWORKSPACE/$DIST/$DPKG_NAME . +cp $DPKG_NAME dockerDir + +cd dockerDir if [ -z "$DOCKER_FROM" ]; then echo "" diff --git a/buildScripts/reportEnvs b/buildScripts/reportEnvs index 7630fc1..b1dd660 100755 --- a/buildScripts/reportEnvs +++ b/buildScripts/reportEnvs @@ -19,12 +19,14 @@ echo " make parallel: [$MAKE_PARALLEL]" echo " distribution: [$BUILD_DIST]" echo "pdf2htmlEX env:" +echo " version: [$PDF2HTMLEX_VERSION]" echo " poppler: [$POPPLER_VERSION]" echo " fontforge: [$FONTFORGE_VERSION]" echo " pdf2htmlEX: [$PDF2HTMLEX_BRANCH]" echo " machine arch: [$MACHINE_ARCH]" echo "release env:" +echo " build date: [$BUILD_DATE]" echo " build time: [$BUILD_TIME]" echo " docker from: [$DOCKER_FROM]" echo " docker name: [$DOCKER_NAME]" From 658b064604a2a4fb5e4fdfcf7c2d3cfb98c524e7 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Fri, 19 Jun 2020 12:55:06 +0100 Subject: [PATCH 50/69] attempt docker build from deb using travisTGclwBM --- buildScripts/travisLinuxDoItAll | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/buildScripts/travisLinuxDoItAll b/buildScripts/travisLinuxDoItAll index e5f6784..12d3b0b 100755 --- a/buildScripts/travisLinuxDoItAll +++ b/buildScripts/travisLinuxDoItAll @@ -34,7 +34,8 @@ export DEBIAN_FRONTEND=noninteractive ./buildScripts/runTests ./buildScripts/createAppImage ./buildScripts/createDebianPackage -./buildScripts/createDockerImageInside -./buildScripts/createDockerImageOutside +./buildScripts/createDockerUbuntuImageFromDeb +#./buildScripts/createDockerImageInside +#./buildScripts/createDockerImageOutside ./buildScripts/uploadGitHubRelease ./buildScripts/uploadDockerImage From 32f37576d59a0a63f98447aeb0e79bcc5de6ea38 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Fri, 19 Jun 2020 17:20:53 +0100 Subject: [PATCH 51/69] removed old createDockerImages added explicit words SUCCESS and FAILURE in browser_tests --- buildScripts/Readme.md | 4 +- buildScripts/buildInstallLocally | 2 +- buildScripts/createDockerImageInside | 46 -------------------- buildScripts/createDockerImageOutside | 62 --------------------------- buildScripts/createImages | 4 +- buildScripts/travisLinuxDoItAll | 6 +-- pdf2htmlEX/test/browser_tests.py | 4 +- 7 files changed, 8 insertions(+), 120 deletions(-) delete mode 100755 buildScripts/createDockerImageInside delete mode 100755 buildScripts/createDockerImageOutside diff --git a/buildScripts/Readme.md b/buildScripts/Readme.md index 082cfd8..ba3f775 100644 --- a/buildScripts/Readme.md +++ b/buildScripts/Readme.md @@ -199,8 +199,8 @@ Typically, most users, will run one of the following "top-level" scripts: `pdf2htmlEX`, installs it and `poppler-data` into a Debian archive (`*.deb`). -- **`createDockerImage`**: Using an already compile version of - `pdf2htmlEX`, installs it and `poppler-data` into a Docker image. +- **`createDockerUbuntuImageFromDeb`**: Installs the Debian archive of + `pdf2htmlEX` created by `createDebianPackage` into a Docker image. - **`getBuildTools`**: Locally `apt` installs all development *tools* required to build `pdf2htmlEX`. diff --git a/buildScripts/buildInstallLocally b/buildScripts/buildInstallLocally index 11c0d69..bd7a49c 100755 --- a/buildScripts/buildInstallLocally +++ b/buildScripts/buildInstallLocally @@ -50,7 +50,7 @@ set -ev #./buildScripts/createAppImage -#./buildScripts/createDockerImage +#./buildScripts/createDockerUbuntuImageFromDeb #./buildScripts/uploadGitHubRelease diff --git a/buildScripts/createDockerImageInside b/buildScripts/createDockerImageInside deleted file mode 100755 index dcd602e..0000000 --- a/buildScripts/createDockerImageInside +++ /dev/null @@ -1,46 +0,0 @@ -#!/bin/bash - -# This bash script creates a docker image from an existing pdf2htmlEX -# -# This is the part which can run *inside* a docker container. - -echo "" -echo "-------------------------------------------------------------------" -echo "CREATING pdf2htmlEX Docker Image (run inside a docker container)" -echo "-------------------------------------------------------------------" -echo "" - -# Collect everything that will be needed... - -source buildScripts/reSourceVersionEnvs - -source buildScripts/dockerFunctions - -set -ev - -mkdir -p imageBuild/dockerDir - -cd pdf2htmlEX/build - -sudo rm -rf install_manifest.txt - -make install DESTDIR=../../imageBuild/dockerDir - -cd ../../poppler-data - -make install \ - prefix=$PDF2HTMLEX_PREFIX \ - datadir=$PDF2HTMLEX_PREFIX/share/pdf2htmlEX \ - DESTDIR=../imageBuild/dockerDir - -cd ../imageBuild/dockerDir - -copy_deps - -# Make sure directories can be traversed by nobody -# -#find . -type d -exec chmod 755 {} \; -# -# Make sure files can be read by nobody -# -#find . -type f -exec chmod 644 {} \; diff --git a/buildScripts/createDockerImageOutside b/buildScripts/createDockerImageOutside deleted file mode 100755 index 390062c..0000000 --- a/buildScripts/createDockerImageOutside +++ /dev/null @@ -1,62 +0,0 @@ -#!/bin/bash - -# This bash script creates a docker image from an existing pdf2htmlEX -# -# This is the part which must be run *outside* of any docker container. - -echo "" -echo "-------------------------------------------------------------------" -echo "CREATING pdf2htmlEX Docker Image (run outside any docker container)" -echo "-------------------------------------------------------------------" -echo "" - -# Collect everything that will be needed... - -source buildScripts/reSourceVersionEnvs - -source buildScripts/dockerFunctions - -set -ev - -mkdir -p imageBuild/dockerDir - -cd imageBuild/dockerDir - -if [ -z "$DOCKER_FROM" ]; then - echo "" - read -p "Enter the docker image for the 'from' base: " DOCKER_FROM - echo "" - if [ -z "$DOCKER_FROM" ]; then - echo "DOCKER_FROM not set... so we can not build the docker image" - exit 1 - fi -fi - -if [ -z "$DOCKER_USERNAME" ]; then - echo "" - read -p "Enter a docker username: " DOCKER_USERNAME - echo "" - if [ -z "$DOCKER_USERNAME" ]; then - echo "DOCKER_USERNAME not set... so we can not build the docker image" - exit 1 - fi -fi - -export DOCKER_NAME="$DOCKER_USERNAME/pdf2htmlex:$PDF2HTMLEX_BRANCH-$BUILD_TIME" - -echo "export DOCKER_FROM=\"$DOCKER_FROM\"" >> ../../buildScripts/reSourceVersionEnvs -echo "export DOCKER_USERNAME=\"$DOCKER_USERNAME\"" >> ../../buildScripts/reSourceVersionEnvs -echo "export DOCKER_NAME=\"$DOCKER_NAME\"" >> ../../buildScripts/reSourceVersionEnvs - -delete_blacklisted - -cat > Dockerfile < Date: Sat, 20 Jun 2020 09:11:30 +0000 Subject: [PATCH 52/69] first attempt at integration of build on Alpine Linx --- buildScripts/Readme.md | 16 ++-- buildScripts/buildFontforge | 7 +- buildScripts/buildInstallLocally | 4 +- buildScripts/buildInstallLocallyAlpine | 57 +++++++++++++++ buildScripts/buildPdf2htmlEX | 5 +- buildScripts/buildPoppler | 4 +- buildScripts/createAppImage | 7 +- buildScripts/createDebianPackage | 7 +- buildScripts/createDockerAlpineImageInside | 37 ++++++++++ buildScripts/createDockerAlpineImageOutside | 77 ++++++++++++++++++++ buildScripts/createDockerUbuntuImageFromDeb | 10 ++- buildScripts/createImages | 4 +- buildScripts/getBuildTools | 6 +- buildScripts/getBuildToolsAlpine | 30 ++++++++ buildScripts/getDevLibraries | 6 +- buildScripts/getDevLibrariesAlpine | 23 ++++++ buildScripts/getFontforge | 7 +- buildScripts/getPoppler | 7 +- buildScripts/installPdf2htmlEX | 2 +- buildScripts/listFilesByChangeTime | 2 +- buildScripts/reportEnvs | 7 +- buildScripts/runTests | 9 ++- buildScripts/travisLinuxDoItAll | 4 +- buildScripts/uploadDockerImage | 7 +- buildScripts/uploadGitHubRelease | 12 +-- buildScripts/uploadGitHubReleaseDSL | 10 +-- buildScripts/uploadImages | 4 +- buildScripts/versionEnvs | 21 ++++-- pdf2htmlEX/CMakeLists.txt | 9 +++ pdf2htmlEX/test/README.md | 2 +- pdf2htmlEX/test/installAutomaticTestSoftware | 4 +- pdf2htmlEX/test/installManualTestSoftware | 4 +- pdf2htmlEX/test/produceHtmlForBrowserTests | 4 +- pdf2htmlEX/test/runLocalBrowserTests | 4 +- pdf2htmlEX/test/runLocalTestsBash | 7 -- pdf2htmlEX/test/runLocalTestsPython | 4 +- pdf2htmlEX/test/runLocalTestsShell | 7 ++ pdf2htmlEX/test/testOutput | 6 +- 38 files changed, 348 insertions(+), 95 deletions(-) create mode 100755 buildScripts/buildInstallLocallyAlpine create mode 100755 buildScripts/createDockerAlpineImageInside create mode 100755 buildScripts/createDockerAlpineImageOutside create mode 100755 buildScripts/getBuildToolsAlpine create mode 100755 buildScripts/getDevLibrariesAlpine delete mode 100755 pdf2htmlEX/test/runLocalTestsBash create mode 100755 pdf2htmlEX/test/runLocalTestsShell diff --git a/buildScripts/Readme.md b/buildScripts/Readme.md index ba3f775..144c71d 100644 --- a/buildScripts/Readme.md +++ b/buildScripts/Readme.md @@ -4,7 +4,7 @@ Because of its intimate use of *specific* versions of both Poppler and FontForge, cleanly building `pdf2htmlEX` is rather more complex than normal. -The (bash) scripts in this directory help automate this mutli-stage +The (shell) scripts in this directory help automate this mutli-stage process. For all but the most experienced programmers, we *strongly* encourage you @@ -111,12 +111,12 @@ However, to get the matched versions of Poppler and FontForge and then compile them statically, *our* build process becomes much more complex than a "simple", `configure, make, make install` cycle. -Hence this directory has a large number of bash scripts each of which +Hence this directory has a large number of shell scripts each of which automate one simple step in the overall our build process. ## The gory details ... -The bash scripts in this directory automate the download, build, install, +The shell scripts in this directory automate the download, build, install, test and upload steps required to provide a complete build/test/release cycle of `pdf2htmlEX`. @@ -263,7 +263,7 @@ Typically, most users, will run one of the following "top-level" scripts: for a standard build of `pdf2htmlEX`. Changes in this script effect *all* of the other build scripts. -- **`reSourceVersionEnvs`**: This bash script is automatically generated +- **`reSourceVersionEnvs`**: This shell script is automatically generated by the build scripts as they are run. It records the values of all important environment variables required by the buildScripts. It is typcically `source`d by each script before it preforms any actions. @@ -272,24 +272,24 @@ Typically, most users, will run one of the following "top-level" scripts: console. This script is used by the top-level scripts to ensure the current environment variables are listed before each build. -- **`uploadGitHubReleaseDSL`**: A collection of bash functions used by the +- **`uploadGitHubReleaseDSL`**: A collection of shell functions used by the `uploadGitHubRelease` script to automate the upload of release artefacts. - **`uploadGitHubReleaseMessage`**: The contents of this *text* file is used by the `uploadGitHubRelease` script as the contents of the release message, as visible to the user, for the 'continuous' release section. -- **`dockerFunctions`**: A collection of bash functions used by the +- **`dockerFunctions`**: A collection of shell functions used by the `uploadDockerImage` script to automate the upload of the Docker images to Docker Hub. -- **`listFilesByChangeTime`**: A simple bash script which lists the files +- **`listFilesByChangeTime`**: A simple shell script which lists the files in the buildScripts directory by most recently changed files first. - **`Readme.md`**: This read me file. ## Yet more details? -The various bash script files are meant to be fairly readable. They +The various shell script files are meant to be fairly readable. They contain additional comments about what each step is meant to be doing. diff --git a/buildScripts/buildFontforge b/buildScripts/buildFontforge index 26cf38a..e58d495 100755 --- a/buildScripts/buildFontforge +++ b/buildScripts/buildFontforge @@ -1,8 +1,9 @@ -#!/bin/bash +#!/bin/sh -# This bash script build FontForge +# This shell script build FontForge -source buildScripts/reSourceVersionEnvs +# source buildScripts/reSourceVersionEnvs +. buildScripts/reSourceVersionEnvs echo "" echo "-------------------------------------------------------------------" diff --git a/buildScripts/buildInstallLocally b/buildScripts/buildInstallLocally index bd7a49c..000dd6b 100755 --- a/buildScripts/buildInstallLocally +++ b/buildScripts/buildInstallLocally @@ -1,6 +1,6 @@ -#!/bin/bash +#!/bin/sh -# This bash script builds the complete pdf2htmlEX application LOCALLY +# This shell script builds the complete pdf2htmlEX application LOCALLY # (It does not create the AppImage or Docker images) # Adjust the following two environment variables to suit your needs diff --git a/buildScripts/buildInstallLocallyAlpine b/buildScripts/buildInstallLocallyAlpine new file mode 100755 index 0000000..8fe6d2f --- /dev/null +++ b/buildScripts/buildInstallLocallyAlpine @@ -0,0 +1,57 @@ +#!/bin/sh + +# This shell script builds the complete pdf2htmlEX application LOCALLY +# (It does not create the AppImage or Docker images) + +# Adjust the following two environment variables to suit your needs +# +export UNATTENDED="--assume-yes" +export MAKE_PARALLEL="-j $(nproc)" + +# choose one of the following... +# +# export PDF2HTMLEX_BRANCH="<>" +export PDF2HTMLEX_BRANCH="$(git rev-parse --abbrev-ref HEAD)" + +# The following environment variable determines where the poppler, +# poppler-data, fontforge and pdf2htmlEX packages are installed. +# CHANGE IT TO SUIT YOUR NEEDS: +# +export PDF2HTMLEX_PREFIX=/usr/local + +# Ensure all Apt packages are installed with no user interaction +# +export DEBIAN_FRONTEND=noninteractive + +set -ev + +################ +# do the build + +./buildScripts/versionEnvs + +./buildScripts/reportEnvs + +./buildScripts/getBuildToolsAlpine + +./buildScripts/getDevLibrariesAlpine + +./buildScripts/getPoppler + +./buildScripts/buildPoppler + +./buildScripts/getFontforge + +./buildScripts/buildFontforge + +./buildScripts/buildPdf2htmlEX + +./buildScripts/installPdf2htmlEX + +#./buildScripts/createAppImage + +#./buildScripts/createDockerUbuntuImageFromDeb + +#./buildScripts/uploadGitHubRelease + +#./buildScripts/uploadDockerImage diff --git a/buildScripts/buildPdf2htmlEX b/buildScripts/buildPdf2htmlEX index 5a048bb..daa62a3 100755 --- a/buildScripts/buildPdf2htmlEX +++ b/buildScripts/buildPdf2htmlEX @@ -1,6 +1,6 @@ -#!/bin/bash +#!/bin/sh -# This bash script builds pdf2htmlEX +# This shell script builds pdf2htmlEX echo "" echo "-------------------------------------------------------------------" @@ -12,6 +12,7 @@ echo "" set -ev cd pdf2htmlEX +rm -rf build mkdir build cd build cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$PDF2HTMLEX_PREFIX .. diff --git a/buildScripts/buildPoppler b/buildScripts/buildPoppler index 3253c75..4583ccb 100755 --- a/buildScripts/buildPoppler +++ b/buildScripts/buildPoppler @@ -1,6 +1,6 @@ -#!/bin/bash +#!/bin/sh -# This bash script builds the latest poppler +# This shell script builds the latest poppler echo "" echo "-------------------------------------------------------------------" diff --git a/buildScripts/createAppImage b/buildScripts/createAppImage index 8c33d52..c2e2fa7 100755 --- a/buildScripts/createAppImage +++ b/buildScripts/createAppImage @@ -1,8 +1,9 @@ -#!/bin/bash +#!/bin/sh -# This bash script creates an AppImage for pdf2htmlEX +# This shell script creates an AppImage for pdf2htmlEX -source ./buildScripts/reSourceVersionEnvs +# source ./buildScripts/reSourceVersionEnvs +. ./buildScripts/reSourceVersionEnvs echo "" echo "-------------------------------------------------------------------" diff --git a/buildScripts/createDebianPackage b/buildScripts/createDebianPackage index 1dbff39..64b0186 100755 --- a/buildScripts/createDebianPackage +++ b/buildScripts/createDebianPackage @@ -1,8 +1,9 @@ -#!/bin/bash +#!/bin/sh -# This bash script creates a (binary) Debian Package Archive for pdf2htmlEX +# This shell script creates a (binary) Debian Package Archive for pdf2htmlEX -source ./buildScripts/reSourceVersionEnvs +# source ./buildScripts/reSourceVersionEnvs +. ./buildScripts/reSourceVersionEnvs echo "" echo "-------------------------------------------------------------------" diff --git a/buildScripts/createDockerAlpineImageInside b/buildScripts/createDockerAlpineImageInside new file mode 100755 index 0000000..c2ba067 --- /dev/null +++ b/buildScripts/createDockerAlpineImageInside @@ -0,0 +1,37 @@ +#!/bin/sh + +# This shell script creates a docker image from an existing pdf2htmlEX +# +# This is the part which can run *inside* a docker container. + +echo "" +echo "-------------------------------------------------------------------" +echo "CREATING pdf2htmlEX Docker Image (run inside a docker container)" +echo " (based on Alpine linux $BUILD_DIST) +echo "-------------------------------------------------------------------" +echo "" + +# Collect everything that will be needed... + +# source buildScripts/reSourceVersionEnvs +. buildScripts/reSourceVersionEnvs + +# source buildScripts/dockerFunctions +. buildScripts/dockerFunctions + +set -ev + +mkdir -p imageBuild/dockerDir + +cd pdf2htmlEX/build + +sudo rm -rf install_manifest.txt + +make install DESTDIR=../../imageBuild/dockerDir + +cd ../../poppler-data + +make install \ + prefix=$PDF2HTMLEX_PREFIX \ + datadir=$PDF2HTMLEX_PREFIX/share/pdf2htmlEX \ + DESTDIR=../imageBuild/dockerDir diff --git a/buildScripts/createDockerAlpineImageOutside b/buildScripts/createDockerAlpineImageOutside new file mode 100755 index 0000000..30fe2f1 --- /dev/null +++ b/buildScripts/createDockerAlpineImageOutside @@ -0,0 +1,77 @@ +#!/bin/sh + +# This shell script creates a docker image from an existing pdf2htmlEX +# +# This is the part which must be run *outside* of any docker container. + +echo "" +echo "-------------------------------------------------------------------" +echo "CREATING pdf2htmlEX Docker Image (run outside any docker container)" +echo " (based on Alpine linux $BUILD_DIST) +echo "-------------------------------------------------------------------" +echo "" + +# Collect everything that will be needed... + +# source buildScripts/reSourceVersionEnvs +. buildScripts/reSourceVersionEnvs + +# source buildScripts/dockerFunctions +. buildScripts/dockerFunctions + +set -ev + +mkdir -p imageBuild/dockerDir + +cd imageBuild/dockerDir + +if [ -z "$DOCKER_FROM" ]; then + echo "" + read -p "Enter the docker image for the 'from' base: " DOCKER_FROM + echo "" + if [ -z "$DOCKER_FROM" ]; then + echo "DOCKER_FROM not set... so we can not build the docker image" + exit 1 + fi +fi + +if [ -z "$DOCKER_USERNAME" ]; then + echo "" + read -p "Enter a docker username: " DOCKER_USERNAME + echo "" + if [ -z "$DOCKER_USERNAME" ]; then + echo "DOCKER_USERNAME not set... so we can not build the docker image" + exit 1 + fi +fi + +export DOCKER_NAME="$DOCKER_USERNAME/pdf2htmlex:$PDF2HTMLEX_BRANCH-$BUILD_TIME" + +echo "export DOCKER_FROM=\"$DOCKER_FROM\"" >> ../../buildScripts/reSourceVersionEnvs +echo "export DOCKER_USERNAME=\"$DOCKER_USERNAME\"" >> ../../buildScripts/reSourceVersionEnvs +echo "export DOCKER_NAME=\"$DOCKER_NAME\"" >> ../../buildScripts/reSourceVersionEnvs + +delete_blacklisted + +cat > Dockerfile <> buildScripts/reSourceV set -ev -# This bash script runs the pdf2htmlEX tests +# This shell script runs the pdf2htmlEX tests cd pdf2htmlEX/test @@ -21,7 +22,7 @@ cd pdf2htmlEX/test # ./installAutomaticTestSoftware -./runLocalTestsBash +./runLocalTestsShell ./runLocalBrowserTests diff --git a/buildScripts/travisLinuxDoItAll b/buildScripts/travisLinuxDoItAll index 063d5b3..6ac8028 100755 --- a/buildScripts/travisLinuxDoItAll +++ b/buildScripts/travisLinuxDoItAll @@ -1,6 +1,6 @@ -#!/bin/bash +#!/bin/sh -# This bash script builds everyting on an TravisCI Linux (Ubunutu) worker +# This shell script builds everyting on an TravisCI Linux (Ubunutu) worker set -ev diff --git a/buildScripts/uploadDockerImage b/buildScripts/uploadDockerImage index 13fe66f..48cb0ec 100755 --- a/buildScripts/uploadDockerImage +++ b/buildScripts/uploadDockerImage @@ -1,6 +1,6 @@ -#!/bin/bash +#!/bin/sh -# This bash script uploads the pdf2htmlEX docker image to docker hub +# This shell script uploads the pdf2htmlEX docker image to docker hub # # We EXPECT the following environment variables to be set: # DOCKER_USERNAME @@ -9,7 +9,8 @@ # DOCKER_PASSWORD (if not set you will be asked for your password) # -source ./buildScripts/reSourceVersionEnvs +# source ./buildScripts/reSourceVersionEnvs +. ./buildScripts/reSourceVersionEnvs echo "" echo "-------------------------------------------------------------------" diff --git a/buildScripts/uploadGitHubRelease b/buildScripts/uploadGitHubRelease index ad4d690..ffde585 100755 --- a/buildScripts/uploadGitHubRelease +++ b/buildScripts/uploadGitHubRelease @@ -1,6 +1,6 @@ -#!/bin/bash +#!/bin/sh -# This bash script uploads the pdf2htmlEX release artefacts +# This shell script uploads the pdf2htmlEX release artefacts # # We EXPECT the following environment variables to be set: # GITHUB_TOKEN @@ -10,9 +10,11 @@ # DOCKER_PASSWORD (if not set you will be asked for your password) # -source ./buildScripts/reSourceVersionEnvs -cp ./buildScripts/reSourceVersionEnvs imageBuild/buildInfo.sh -source ./buildScripts/uploadGitHubReleaseDSL +# source ./buildScripts/reSourceVersionEnvs +. ./buildScripts/reSourceVersionEnvs +cp ./buildScripts/reSourceVersionEnvs imageBuild/buildInfo.sh +. ./buildScripts/uploadGitHubReleaseDSL +# source ./buildScripts/uploadGitHubReleaseDSL echo "" echo "-------------------------------------------------------------------" diff --git a/buildScripts/uploadGitHubReleaseDSL b/buildScripts/uploadGitHubReleaseDSL index 3a0207a..eb226d6 100755 --- a/buildScripts/uploadGitHubReleaseDSL +++ b/buildScripts/uploadGitHubReleaseDSL @@ -1,8 +1,8 @@ -#!/usr/bin/env bash +#!/usr/bin/env sh -# This bash script is a collection of functions to interact with the GitHub -# release and git/tag APIs. It essentially provides a bash based DSL for -# uploading GitHub release artefacts. +# This shell script is a collection of functions to interact with the GitHub +# release and git/tag APIs. It essentially provides a shell script based DSL +# for uploading GitHub release artefacts. # NOTE-1: These functions make use of your ".netrc" file to authenticate # with GitHub. In order to use these functions, make sure you have **both** @@ -171,7 +171,7 @@ function uploadAnAsset { echo "" } -# The above has been heavily modified into a collection of bash functions +# The above has been heavily modified into a collection of shell functions # by Stephen Gaito working on the pdf2htmlEX project. # # It has been based upon the code in: diff --git a/buildScripts/uploadImages b/buildScripts/uploadImages index 7adbe9c..28760a0 100755 --- a/buildScripts/uploadImages +++ b/buildScripts/uploadImages @@ -1,6 +1,6 @@ -#!/bin/bash +#!/bin/sh -# This bash script uploads already existing pdf2htmlEX AppImage and Docker +# This shell script uploads already existing pdf2htmlEX AppImage and Docker # Images ################ diff --git a/buildScripts/versionEnvs b/buildScripts/versionEnvs index d8c1da6..637a3b7 100755 --- a/buildScripts/versionEnvs +++ b/buildScripts/versionEnvs @@ -1,6 +1,6 @@ -#!/bin/bash +#!/bin/sh -# This bash script exports environment variables for the latest software +# This shell script exports environment variables for the latest software # versions # see: https://poppler.freedesktop.org/releases.html @@ -42,11 +42,18 @@ if [ -z "$PDF2HTMLEX_BRANCH" ]; then fi fi -cat /etc/lsb-release - -source /etc/lsb-release - -export BUILD_DIST=$DISTRIB_CODENAME +if test -r /etc/lsb-release ; then + cat /etc/lsb-release + # source /etc/lsb-release + . /etc/lsb-release + export BUILD_DIST=$DISTRIB_CODENAME +elif test -r /etc/alpine-release ; then + cat /etc/alpine-release + export BUILD_DIST=$(cat /etc/alpine-release) +else + echo "FAILURE: could not determine release" + exit -1 +fi export BUILD_DATE="$(date +%Y%m%d)" diff --git a/pdf2htmlEX/CMakeLists.txt b/pdf2htmlEX/CMakeLists.txt index ab3c61a..218c54d 100644 --- a/pdf2htmlEX/CMakeLists.txt +++ b/pdf2htmlEX/CMakeLists.txt @@ -79,9 +79,18 @@ set(FONTFORGE_LIBRARIES ${FONTFORGE_LIBRARIES} ${CMAKE_SOURCE_DIR}/../fontforge/build/lib/libfontforge.a ) +# If we are using Alpine Linux then we need to add -lintl +# +if (EXISTS /usr/lib/libintl.so ) + set(LIB_INTL_LIBRARIES -lintl ) +else () + set(LIB_INTL_LIBRARIES "" ) +endif() + set(PDF2HTMLEX_LIBS ${PDF2HTMLEX_LIBS} ${POPPLER_LIBRARIES} ${FONTFORGE_LIBRARIES} + ${LIB_INTL_LIBRARIES} ${CAIRO_LIBRARIES} -ljpeg -lpng diff --git a/pdf2htmlEX/test/README.md b/pdf2htmlEX/test/README.md index 7b8f708..24a0209 100644 --- a/pdf2htmlEX/test/README.md +++ b/pdf2htmlEX/test/README.md @@ -19,7 +19,7 @@ and needs to be fixed. Rerun `runLocalBrowserTests` and use the ## Running tests -There are three bash scripts which automate the running of a given +There are three shell scripts which automate the running of a given collection of tests: 1. **runLocalTests** runs a simple collection of tests which do not diff --git a/pdf2htmlEX/test/installAutomaticTestSoftware b/pdf2htmlEX/test/installAutomaticTestSoftware index 2b53ad2..d1b73dd 100755 --- a/pdf2htmlEX/test/installAutomaticTestSoftware +++ b/pdf2htmlEX/test/installAutomaticTestSoftware @@ -1,8 +1,8 @@ -#!/bin/bash +#!/bin/sh set -ev -# This bash script installs all local software required to run the +# This shell script installs all local software required to run the # pdf2htmlEX tests export DEBIAN_FRONTEND=noninteractive diff --git a/pdf2htmlEX/test/installManualTestSoftware b/pdf2htmlEX/test/installManualTestSoftware index d7c8ab3..5dc3e8a 100755 --- a/pdf2htmlEX/test/installManualTestSoftware +++ b/pdf2htmlEX/test/installManualTestSoftware @@ -1,6 +1,6 @@ -#!/bin/bash +#!/bin/sh -# This bash script installs all local software required to run the +# This shell script installs all local software required to run the # pdf2htmlEX tests export DEBIAN_FRONTEND=noninteractive diff --git a/pdf2htmlEX/test/produceHtmlForBrowserTests b/pdf2htmlEX/test/produceHtmlForBrowserTests index 6225aab..c0979a6 100755 --- a/pdf2htmlEX/test/produceHtmlForBrowserTests +++ b/pdf2htmlEX/test/produceHtmlForBrowserTests @@ -1,6 +1,6 @@ -#!/bin/bash +#!/bin/sh -# This bash script walks through the browser_tests directory running +# This shell script walks through the browser_tests directory running # pdf2htmlEX on each *.pdf file. # This is how we run pdf2htmlEX on a particular file, and arguments. diff --git a/pdf2htmlEX/test/runLocalBrowserTests b/pdf2htmlEX/test/runLocalBrowserTests index f65af7b..f27cae2 100755 --- a/pdf2htmlEX/test/runLocalBrowserTests +++ b/pdf2htmlEX/test/runLocalBrowserTests @@ -1,6 +1,6 @@ -#!/bin/bash +#!/bin/sh -# This bash script runs the local browser tests +# This shell script runs the local browser tests # We start by running a virtual frame buffer as display 99.0 # diff --git a/pdf2htmlEX/test/runLocalTestsBash b/pdf2htmlEX/test/runLocalTestsBash deleted file mode 100755 index f3bf575..0000000 --- a/pdf2htmlEX/test/runLocalTestsBash +++ /dev/null @@ -1,7 +0,0 @@ -#!/bin/bash - -set -ev - -# This bash script runs the (simple non-browser) tests - -./testOutput diff --git a/pdf2htmlEX/test/runLocalTestsPython b/pdf2htmlEX/test/runLocalTestsPython index f8db9c7..8f0fa85 100755 --- a/pdf2htmlEX/test/runLocalTestsPython +++ b/pdf2htmlEX/test/runLocalTestsPython @@ -1,7 +1,7 @@ -#!/bin/bash +#!/bin/sh set -ev -# This bash script runs the (simple non-browser) tests +# This shell script runs the (simple non-browser) tests python3 test_output.py diff --git a/pdf2htmlEX/test/runLocalTestsShell b/pdf2htmlEX/test/runLocalTestsShell new file mode 100755 index 0000000..692245a --- /dev/null +++ b/pdf2htmlEX/test/runLocalTestsShell @@ -0,0 +1,7 @@ +#!/bin/sh + +set -ev + +# This shell script runs the (simple non-browser) tests + +./testOutput diff --git a/pdf2htmlEX/test/testOutput b/pdf2htmlEX/test/testOutput index c18ebdb..8e11fd3 100755 --- a/pdf2htmlEX/test/testOutput +++ b/pdf2htmlEX/test/testOutput @@ -1,9 +1,9 @@ -#!/bin/bash +#!/bin/sh -# This bash script checks that pdf2htmlEX does not crash, and produces +# This shell script checks that pdf2htmlEX does not crash, and produces # correct files. Do not check the content of the files. -# We use a bash script to implement the python3 test_output.py so that we +# We use a shell script to implement the python3 test_output.py so that we # do not need to install any extra packages to test this functionality. # This is how we run pdf2htmlEX on a particular file, and arguments. From 8a04d6a19223305a7c60d0cb56fced8fa980d1fa Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sat, 20 Jun 2020 10:13:26 +0100 Subject: [PATCH 53/69] added browser_tests SUCCESS if dff_bbox is not None but basefilename is test_fail --- pdf2htmlEX/test/browser_tests.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/pdf2htmlEX/test/browser_tests.py b/pdf2htmlEX/test/browser_tests.py index 5d1a242..e09c54a 100644 --- a/pdf2htmlEX/test/browser_tests.py +++ b/pdf2htmlEX/test/browser_tests.py @@ -95,6 +95,8 @@ class BrowserTests(Common): if diff_bbox is None: print(" SUCCESS: ", basefilename) + elif basefilename == "test_fail" : + print(" SUCCESS: ", basefilename) else: print(" FAILURE: ", basefilename, " diff bounding box: ", diff_bbox, " should be None") diff_size = (diff_bbox[2] - diff_bbox[0]) * (diff_bbox[3] - diff_bbox[1]) From 98acb27bfef531045e1a2f28c6706b81d3eca388 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sat, 20 Jun 2020 10:31:35 +0100 Subject: [PATCH 54/69] begun to explicitly name apt and ubuntu buildScripts --- buildScripts/{buildInstallLocally => buildInstallLocallyApt} | 4 ++-- buildScripts/{getBuildTools => getBuildToolsApt} | 0 buildScripts/{getDevLibraries => getDevLibrariesApt} | 0 3 files changed, 2 insertions(+), 2 deletions(-) rename buildScripts/{buildInstallLocally => buildInstallLocallyApt} (94%) rename buildScripts/{getBuildTools => getBuildToolsApt} (100%) rename buildScripts/{getDevLibraries => getDevLibrariesApt} (100%) diff --git a/buildScripts/buildInstallLocally b/buildScripts/buildInstallLocallyApt similarity index 94% rename from buildScripts/buildInstallLocally rename to buildScripts/buildInstallLocallyApt index 000dd6b..2dcded2 100755 --- a/buildScripts/buildInstallLocally +++ b/buildScripts/buildInstallLocallyApt @@ -32,9 +32,9 @@ set -ev ./buildScripts/reportEnvs -./buildScripts/getBuildTools +./buildScripts/getBuildToolsApt -./buildScripts/getDevLibraries +./buildScripts/getDevLibrariesApt ./buildScripts/getPoppler diff --git a/buildScripts/getBuildTools b/buildScripts/getBuildToolsApt similarity index 100% rename from buildScripts/getBuildTools rename to buildScripts/getBuildToolsApt diff --git a/buildScripts/getDevLibraries b/buildScripts/getDevLibrariesApt similarity index 100% rename from buildScripts/getDevLibraries rename to buildScripts/getDevLibrariesApt From c09362078d5af29f6d349b6c66fa0481c218b62e Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sat, 20 Jun 2020 11:52:17 +0100 Subject: [PATCH 55/69] continued work on making ubuntu and alpine builds distinct --- ...utomaticTestSoftware => installAutomaticTestSoftwareApt} | 0 pdf2htmlEX/test/produceHtmlForBrowserTests | 4 ++-- pdf2htmlEX/test/testOutput | 6 +++--- 3 files changed, 5 insertions(+), 5 deletions(-) rename pdf2htmlEX/test/{installAutomaticTestSoftware => installAutomaticTestSoftwareApt} (100%) diff --git a/pdf2htmlEX/test/installAutomaticTestSoftware b/pdf2htmlEX/test/installAutomaticTestSoftwareApt similarity index 100% rename from pdf2htmlEX/test/installAutomaticTestSoftware rename to pdf2htmlEX/test/installAutomaticTestSoftwareApt diff --git a/pdf2htmlEX/test/produceHtmlForBrowserTests b/pdf2htmlEX/test/produceHtmlForBrowserTests index c0979a6..448a35f 100755 --- a/pdf2htmlEX/test/produceHtmlForBrowserTests +++ b/pdf2htmlEX/test/produceHtmlForBrowserTests @@ -5,7 +5,7 @@ # This is how we run pdf2htmlEX on a particular file, and arguments. # -function runPdf2htmlEX { +runPdf2htmlEX() { pdfFileName=$1 htmlFileName=$(echo $pdfFileName | cut -d'.' -f1).html arguments=$2 @@ -39,7 +39,7 @@ function runPdf2htmlEX { # and '#TEST_IGNORE_END' (we pipe the file in via stdin and save it via # stdout) # -function copy_TEST_IGNORE_file { +copy_TEST_IGNORE_file() { skipLine=echo while IFS= read -r line ; do if echo $line | grep -q "^#TEST_IGNORE_BEGIN" ; then diff --git a/pdf2htmlEX/test/testOutput b/pdf2htmlEX/test/testOutput index 8e11fd3..3a44f25 100755 --- a/pdf2htmlEX/test/testOutput +++ b/pdf2htmlEX/test/testOutput @@ -8,7 +8,7 @@ # This is how we run pdf2htmlEX on a particular file, and arguments. # -function runPdf2htmlEX { +runPdf2htmlEX() { # collect the arguments # export LAST_TEST_NAME="$*" @@ -36,7 +36,7 @@ function runPdf2htmlEX { # This is how we test for expected output files # -function hasExpectedFiles { +hasExpectedFiles() { filesFound="true" for anExpectedFile in $1 ; do if ! test -r $PDF2HTMLEX_TMPDIR/$anExpectedFile ; then @@ -57,7 +57,7 @@ function hasExpectedFiles { # and '#TEST_IGNORE_END' (we pipe the file in via stdin and save it via # stdout) # -function copy_TEST_IGNORE_file { +copy_TEST_IGNORE_file() { skipLine=echo while IFS= read -r line ; do if echo $line | grep -q "^#TEST_IGNORE_BEGIN" ; then From 310bbccfe19679e3dc4e3c73aa72c8989b970796 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sat, 20 Jun 2020 12:29:01 +0100 Subject: [PATCH 56/69] corrected installation of automaice test software --- pdf2htmlEX/test/installAutomaticTestSoftwareApt | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/pdf2htmlEX/test/installAutomaticTestSoftwareApt b/pdf2htmlEX/test/installAutomaticTestSoftwareApt index d1b73dd..fd41a7f 100755 --- a/pdf2htmlEX/test/installAutomaticTestSoftwareApt +++ b/pdf2htmlEX/test/installAutomaticTestSoftwareApt @@ -20,7 +20,8 @@ sudo apt -y install \ # Now get the geckodriver for firefox (as required by selenium) # -pushd /tmp +oldPWD=$(pwd) +cd /tmp # wget https://github.com/mozilla/geckodriver/releases/download/v0.26.0/geckodriver-v0.26.0-linux64.tar.gz # @@ -28,7 +29,7 @@ tar xvf geckodriver-v0.26.0-linux64.tar.gz # sudo mv geckodriver /usr/local/bin # -popd +cd $oldPWD # Now make sure all python packages exist (install into the local user's # PyPI archive) From 8f19c49a7ab574afc6be5f850599ae2ce3897845 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sat, 20 Jun 2020 13:04:56 +0100 Subject: [PATCH 57/69] corrected travis error --- buildScripts/travisLinuxDoItAll | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/buildScripts/travisLinuxDoItAll b/buildScripts/travisLinuxDoItAll index 6ac8028..4d3e697 100755 --- a/buildScripts/travisLinuxDoItAll +++ b/buildScripts/travisLinuxDoItAll @@ -23,8 +23,8 @@ export DEBIAN_FRONTEND=noninteractive ./buildScripts/versionEnvs ./buildScripts/reportEnvs -./buildScripts/getBuildTools -./buildScripts/getDevLibraries +./buildScripts/getBuildToolsApt +./buildScripts/getDevLibrariesApt ./buildScripts/getPoppler ./buildScripts/buildPoppler ./buildScripts/getFontforge From c235a6ba7d0a55bdfedaf23bbbc52ea4d9e5772a Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sat, 20 Jun 2020 13:12:53 +0100 Subject: [PATCH 58/69] continued corrections of travis build and test --- buildScripts/runTests | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/buildScripts/runTests b/buildScripts/runTests index b31996a..b7d4f8b 100755 --- a/buildScripts/runTests +++ b/buildScripts/runTests @@ -20,7 +20,7 @@ cd pdf2htmlEX/test # The following is only needed for the local browser tests # -./installAutomaticTestSoftware +./installAutomaticTestSoftwareApt ./runLocalTestsShell From 1dab7509c4c0fe9777cfdb63b25855802e3e3e9d Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sat, 20 Jun 2020 16:05:01 +0100 Subject: [PATCH 59/69] added freetype-dev to alpine build --- buildScripts/getDevLibrariesAlpine | 1 + 1 file changed, 1 insertion(+) diff --git a/buildScripts/getDevLibrariesAlpine b/buildScripts/getDevLibrariesAlpine index bd1e208..c9a95e0 100755 --- a/buildScripts/getDevLibrariesAlpine +++ b/buildScripts/getDevLibrariesAlpine @@ -19,5 +19,6 @@ sudo apk add \ gettext \ cairo \ libpng-dev \ + freetype-dev \ libjpeg-turbo-dev \ libxml2-dev From e777c6ef4b359dc856d89cd133fe28101e84dc5b Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sat, 20 Jun 2020 16:09:23 +0100 Subject: [PATCH 60/69] added fontconfig-dev to alpine build --- buildScripts/getDevLibrariesAlpine | 1 + 1 file changed, 1 insertion(+) diff --git a/buildScripts/getDevLibrariesAlpine b/buildScripts/getDevLibrariesAlpine index c9a95e0..a5f7a27 100755 --- a/buildScripts/getDevLibrariesAlpine +++ b/buildScripts/getDevLibrariesAlpine @@ -20,5 +20,6 @@ sudo apk add \ cairo \ libpng-dev \ freetype-dev \ + fontconfig-dev \ libjpeg-turbo-dev \ libxml2-dev From 83b12315b3acd16c4f080a602565d0ff20c8a7bd Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sat, 20 Jun 2020 16:17:03 +0100 Subject: [PATCH 61/69] added gettext-dev to alpine build --- buildScripts/getDevLibrariesAlpine | 1 + 1 file changed, 1 insertion(+) diff --git a/buildScripts/getDevLibrariesAlpine b/buildScripts/getDevLibrariesAlpine index a5f7a27..72b035b 100755 --- a/buildScripts/getDevLibrariesAlpine +++ b/buildScripts/getDevLibrariesAlpine @@ -20,6 +20,7 @@ sudo apk add \ cairo \ libpng-dev \ freetype-dev \ + gettext-dev \ fontconfig-dev \ libjpeg-turbo-dev \ libxml2-dev From 3d5eb634867c2a0ec3e3082e72b48d4ce2ad8a52 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sat, 20 Jun 2020 16:38:46 +0100 Subject: [PATCH 62/69] added glib-dev to alpine build --- buildScripts/getDevLibrariesAlpine | 1 + 1 file changed, 1 insertion(+) diff --git a/buildScripts/getDevLibrariesAlpine b/buildScripts/getDevLibrariesAlpine index 72b035b..f0f8b3c 100755 --- a/buildScripts/getDevLibrariesAlpine +++ b/buildScripts/getDevLibrariesAlpine @@ -21,6 +21,7 @@ sudo apk add \ libpng-dev \ freetype-dev \ gettext-dev \ + glib-dev \ fontconfig-dev \ libjpeg-turbo-dev \ libxml2-dev From e302947e48ccc1f0e5102f17f31475565281ebbb Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sat, 20 Jun 2020 16:54:37 +0100 Subject: [PATCH 63/69] added cairo-dev to alpine build --- buildScripts/getDevLibrariesAlpine | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/buildScripts/getDevLibrariesAlpine b/buildScripts/getDevLibrariesAlpine index f0f8b3c..a8a3c41 100755 --- a/buildScripts/getDevLibrariesAlpine +++ b/buildScripts/getDevLibrariesAlpine @@ -17,7 +17,7 @@ set -ev sudo apk update sudo apk add \ gettext \ - cairo \ + cairo-dev \ libpng-dev \ freetype-dev \ gettext-dev \ From 8455d458ec5c46284ffbc29b3c47f53e110efe2d Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sat, 20 Jun 2020 17:22:46 +0100 Subject: [PATCH 64/69] corrected alpine docker image creation scripts --- buildScripts/createDockerAlpineImageInside | 2 +- buildScripts/createDockerAlpineImageOutside | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/buildScripts/createDockerAlpineImageInside b/buildScripts/createDockerAlpineImageInside index c2ba067..759899c 100755 --- a/buildScripts/createDockerAlpineImageInside +++ b/buildScripts/createDockerAlpineImageInside @@ -7,7 +7,7 @@ echo "" echo "-------------------------------------------------------------------" echo "CREATING pdf2htmlEX Docker Image (run inside a docker container)" -echo " (based on Alpine linux $BUILD_DIST) +echo " (based on Alpine linux $BUILD_DIST)" echo "-------------------------------------------------------------------" echo "" diff --git a/buildScripts/createDockerAlpineImageOutside b/buildScripts/createDockerAlpineImageOutside index 30fe2f1..d1e971c 100755 --- a/buildScripts/createDockerAlpineImageOutside +++ b/buildScripts/createDockerAlpineImageOutside @@ -7,7 +7,7 @@ echo "" echo "-------------------------------------------------------------------" echo "CREATING pdf2htmlEX Docker Image (run outside any docker container)" -echo " (based on Alpine linux $BUILD_DIST) +echo " (based on Alpine linux $BUILD_DIST)" echo "-------------------------------------------------------------------" echo "" From 6913b970d7c63153ceea7af9cd6b4473b45b2b21 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sat, 20 Jun 2020 19:22:42 +0100 Subject: [PATCH 65/69] added additional runtime dependencies of pdf2htmlEX inside Alpine linux --- buildScripts/createDockerAlpineImageOutside | 22 ++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/buildScripts/createDockerAlpineImageOutside b/buildScripts/createDockerAlpineImageOutside index d1e971c..0ad7603 100755 --- a/buildScripts/createDockerAlpineImageOutside +++ b/buildScripts/createDockerAlpineImageOutside @@ -51,22 +51,22 @@ echo "export DOCKER_FROM=\"$DOCKER_FROM\"" >> ../../buildScripts/reSourc echo "export DOCKER_USERNAME=\"$DOCKER_USERNAME\"" >> ../../buildScripts/reSourceVersionEnvs echo "export DOCKER_NAME=\"$DOCKER_NAME\"" >> ../../buildScripts/reSourceVersionEnvs -delete_blacklisted - cat > Dockerfile < Date: Sat, 20 Jun 2020 19:46:28 +0100 Subject: [PATCH 66/69] added gnu-libiconv runtime dependencies of pdf2htmlEX inside Alpine linux --- buildScripts/createDockerAlpineImageOutside | 1 + 1 file changed, 1 insertion(+) diff --git a/buildScripts/createDockerAlpineImageOutside b/buildScripts/createDockerAlpineImageOutside index 0ad7603..17af024 100755 --- a/buildScripts/createDockerAlpineImageOutside +++ b/buildScripts/createDockerAlpineImageOutside @@ -60,6 +60,7 @@ RUN apk update && \ apk add --no-cache \ libstdc++ \ libgcc \ + gnu-libiconv \ gettext \ glib \ freetype \ From f340abb31eb9170e235475a7e690dd043ec52850 Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sun, 21 Jun 2020 07:12:05 +0100 Subject: [PATCH 67/69] updated names of created images add gnu-libiconv-dev to alpine build --- buildScripts/createAppImage | 2 +- buildScripts/createDebianPackage | 2 +- buildScripts/createDockerAlpineImageOutside | 2 +- buildScripts/createDockerUbuntuImageFromDeb | 2 +- buildScripts/getDevLibrariesAlpine | 1 + buildScripts/versionEnvs | 10 ++++++++-- 6 files changed, 13 insertions(+), 6 deletions(-) diff --git a/buildScripts/createAppImage b/buildScripts/createAppImage index c2e2fa7..1a72152 100755 --- a/buildScripts/createAppImage +++ b/buildScripts/createAppImage @@ -16,7 +16,7 @@ set -ev # For appimage output plugin # #export VERSION="$PDF2HTMLEX_BRANCH-$BUILD_TIME" -export APPIMAGE_NAME="pdf2htmlEX-$PDF2HTMLEX_BRANCH-$BUILD_TIME-$BUILD_DIST-$MACHINE_ARCH.AppImage" +export APPIMAGE_NAME="pdf2htmlEX-$PDF2HTMLEX_NAME.AppImage" echo "export APPIMAGE_NAME=\"$APPIMAGE_NAME\"" >> buildScripts/reSourceVersionEnvs diff --git a/buildScripts/createDebianPackage b/buildScripts/createDebianPackage index 64b0186..3c47d5f 100755 --- a/buildScripts/createDebianPackage +++ b/buildScripts/createDebianPackage @@ -13,7 +13,7 @@ echo "" set -ev -export DPKG_NAME="pdf2htmlEX-$PDF2HTMLEX_BRANCH-$BUILD_TIME-$BUILD_DIST-$MACHINE_ARCH.deb" +export DPKG_NAME="pdf2htmlEX-$PDF2HTMLEX_NAME.deb" echo "export DPKG_NAME=\"$DPKG_NAME\"" >> buildScripts/reSourceVersionEnvs diff --git a/buildScripts/createDockerAlpineImageOutside b/buildScripts/createDockerAlpineImageOutside index 17af024..ff6fa44 100755 --- a/buildScripts/createDockerAlpineImageOutside +++ b/buildScripts/createDockerAlpineImageOutside @@ -45,7 +45,7 @@ if [ -z "$DOCKER_USERNAME" ]; then fi fi -export DOCKER_NAME="$DOCKER_USERNAME/pdf2htmlex:$PDF2HTMLEX_BRANCH-$BUILD_TIME" +export DOCKER_NAME="$DOCKER_USERNAME/pdf2htmlex:$PDF2HTMLEX_NAME" echo "export DOCKER_FROM=\"$DOCKER_FROM\"" >> ../../buildScripts/reSourceVersionEnvs echo "export DOCKER_USERNAME=\"$DOCKER_USERNAME\"" >> ../../buildScripts/reSourceVersionEnvs diff --git a/buildScripts/createDockerUbuntuImageFromDeb b/buildScripts/createDockerUbuntuImageFromDeb index c2f0af8..c0e4030 100755 --- a/buildScripts/createDockerUbuntuImageFromDeb +++ b/buildScripts/createDockerUbuntuImageFromDeb @@ -48,7 +48,7 @@ if [ -z "$DOCKER_USERNAME" ]; then fi fi -export DOCKER_NAME="$DOCKER_USERNAME/pdf2htmlex:$PDF2HTMLEX_BRANCH-$BUILD_TIME" +export DOCKER_NAME="$DOCKER_USERNAME/pdf2htmlex:$PDF2HTMLEX_NAME" echo "export DOCKER_FROM=\"$DOCKER_FROM\"" >> ../../buildScripts/reSourceVersionEnvs echo "export DOCKER_USERNAME=\"$DOCKER_USERNAME\"" >> ../../buildScripts/reSourceVersionEnvs diff --git a/buildScripts/getDevLibrariesAlpine b/buildScripts/getDevLibrariesAlpine index a8a3c41..f2a6ec2 100755 --- a/buildScripts/getDevLibrariesAlpine +++ b/buildScripts/getDevLibrariesAlpine @@ -17,6 +17,7 @@ set -ev sudo apk update sudo apk add \ gettext \ + gnu-libiconv-dev \ cairo-dev \ libpng-dev \ freetype-dev \ diff --git a/buildScripts/versionEnvs b/buildScripts/versionEnvs index 637a3b7..00ac4b6 100755 --- a/buildScripts/versionEnvs +++ b/buildScripts/versionEnvs @@ -47,8 +47,10 @@ if test -r /etc/lsb-release ; then # source /etc/lsb-release . /etc/lsb-release export BUILD_DIST=$DISTRIB_CODENAME + export BUILD_OS=$DISTRIB_ID elif test -r /etc/alpine-release ; then cat /etc/alpine-release + export BUILD_OS=alpine export BUILD_DIST=$(cat /etc/alpine-release) else echo "FAILURE: could not determine release" @@ -61,11 +63,15 @@ export BUILD_TIME="$(date +%Y_%m_%d-%H_%M_%S)" export MACHINE_ARCH="$(uname -m)" -echo "export PDF2HTMLEX_VERSION=\"$PDF2HTMLEX_VERSION\"" > buildScripts/reSourceVersionEnvs -echo "export POPPLER_VERSION=\"$POPPLER_VERSION\"" >> buildScripts/reSourceVersionEnvs +export PDF2HTMLEX_NAME=$PDF2HTMLEX_VERSION-$PDF2HTMLEX_BRANCH-$BUILD_DATE-$BUILD_OS-$BUILD_DIST-$MACHINE_ARCH + +echo "export PDF2HTMLEX_NAME=\"$PDF2HTMLEX_NAME\"" > buildScripts/reSourceVersionEnvs +echo "export PDF2HTMLEX_VERSION=\"$PDF2HTMLEX_VERSION\"" >> buildScripts/reSourceVersionEnvs +echo "export POPPLER_VERSION=\"$POPPLER_VERSION\"" >> buildScripts/reSourceVersionEnvs echo "export FONTFORGE_VERSION=\"$FONTFORGE_VERSION\"" >> buildScripts/reSourceVersionEnvs echo "export PDF2HTMLEX_BRANCH=\"$PDF2HTMLEX_BRANCH\"" >> buildScripts/reSourceVersionEnvs echo "export MACHINE_ARCH=\"$MACHINE_ARCH\"" >> buildScripts/reSourceVersionEnvs +echo "export BUILD_OS=\"$BUILD_OS\"" >> buildScripts/reSourceVersionEnvs echo "export BUILD_DIST=\"$BUILD_DIST\"" >> buildScripts/reSourceVersionEnvs echo "export BUILD_DATE=\"$BUILD_DATE\"" >> buildScripts/reSourceVersionEnvs echo "export BUILD_TIME=\"$BUILD_TIME\"" >> buildScripts/reSourceVersionEnvs From 0d2c5048b0dcb9b65eca4868587cfe848946eaac Mon Sep 17 00:00:00 2001 From: Stephen Gaito Date: Sun, 21 Jun 2020 12:00:27 +0100 Subject: [PATCH 68/69] refactored Alpine image creation --- CONTRIBUTING.md | 10 +- INSTALL | 2 +- LICENSE | 2 +- {debian => archive/debian}/changelog | 0 {debian => archive/debian}/compat | 0 {debian => archive/debian}/control | 0 {debian => archive/debian}/copyright | 0 {debian => archive/debian}/dirs | 0 {debian => archive/debian}/pdf2htmlex.NEWS | 0 {debian => archive/debian}/pdf2htmlex.README | 0 {debian => archive/debian}/pdf2htmlex.TODO | 0 {debian => archive/debian}/rules | 0 {debian => archive/debian}/source/format | 0 buildScripts/Readme.md | 91 +++++++++++++------ buildScripts/buildInstallLocallyAlpine | 8 -- buildScripts/buildInstallLocallyApt | 8 -- buildScripts/createAlpineTarFile | 79 ++++++++++++++++ ...ide => createDockerAlpineImageFromTarFile} | 21 +---- buildScripts/createDockerAlpineImageInside | 37 -------- buildScripts/createDockerUbuntuImageFromDeb | 5 +- buildScripts/createImagesAlpine | 19 ++++ .../{createImages => createImagesApt} | 0 buildScripts/dockerFunctions | 59 ------------ pdf2htmlEX/pdf2htmlEX.1.in | 4 +- pdf2htmlEX/share/base.css.in | 2 +- pdf2htmlEX/share/fancy.css.in | 2 +- pdf2htmlEX/share/manifest | 2 +- pdf2htmlEX/share/pdf2htmlEX.js.in | 2 +- pdf2htmlEX/test/test_local_browser.py | 5 +- 29 files changed, 186 insertions(+), 172 deletions(-) rename {debian => archive/debian}/changelog (100%) rename {debian => archive/debian}/compat (100%) rename {debian => archive/debian}/control (100%) rename {debian => archive/debian}/copyright (100%) rename {debian => archive/debian}/dirs (100%) rename {debian => archive/debian}/pdf2htmlex.NEWS (100%) rename {debian => archive/debian}/pdf2htmlex.README (100%) rename {debian => archive/debian}/pdf2htmlex.TODO (100%) rename {debian => archive/debian}/rules (100%) rename {debian => archive/debian}/source/format (100%) create mode 100755 buildScripts/createAlpineTarFile rename buildScripts/{createDockerAlpineImageOutside => createDockerAlpineImageFromTarFile} (81%) delete mode 100755 buildScripts/createDockerAlpineImageInside create mode 100755 buildScripts/createImagesAlpine rename buildScripts/{createImages => createImagesApt} (100%) delete mode 100644 buildScripts/dockerFunctions diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 72508e2..51159c6 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -23,7 +23,7 @@ A few channels are available to reach the developers, please find the most prope ### The Issue Tracker -The [Issue Tracker](https://github.com/coolwanglu/pdf2htmlEX/issues) +The [Issue Tracker](https://github.com/pdf2htmlEX/pdf2htmlEX/issues) is the best way for [bug reports](#bug-reports), [features requests](#feature-requests) @@ -54,7 +54,7 @@ However do not report issues or submit patches there, since it's terrible to kee pdf2htmlEX is mostly written and maintained by 王璐 (Lu Wang). His email and twitter account can be found in -[README.md](https://github.com/coolwanglu/pdf2htmlEX/blob/master/README.md). +[README.md](https://github.com/pdf2htmlEX/pdf2htmlEX/blob/master/README.md). Please post only messages that do not fit into the above channels, otherwise note that he no longer replies with _Please post your question to the mailing list_ or _Please file an issue at GitHub_, consider your message already replied. @@ -73,7 +73,7 @@ follow the following steps to get it resolved as fast as possible: First of all, did you realize that your question might have been already answered in one of the following places? -- [pdf2htmlEX Wiki](https://github.com/coolwanglu/pdf2htmlEX/wiki) +- [pdf2htmlEX Wiki](https://github.com/pdf2htmlEX/pdf2htmlEX/wiki) - The manpage (run `man pdf2htmlEX`) - Old posts in the [mailing list](#the-mailing-list) or the [issue tracker](#the-issue-tracker) - [Google](http://www.google.com/) @@ -93,7 +93,7 @@ Bugs should always be reported to [the Issue Tracker](#the-issue-tracker). Before you report any bug: - Use the latest git version of pdf2htmlEX, since the issue may have been already fixed. - Search for previous issues (open or closed), to make sure that the issue has not been reported before. -- If pdf2htmlEX crashed, take a look at [this article](https://github.com/coolwanglu/pdf2htmlEX/wiki/Troubleshooting-Crashes). +- If pdf2htmlEX crashed, take a look at [this article](https://github.com/pdf2htmlEX/pdf2htmlEX/wiki/Troubleshooting-Crashes). A good bug report shouldn't leave others needing to chase you up for more information. The developers may be very familiar with the code base of pdf2htmlEX, @@ -110,7 +110,7 @@ If you are not sure, please try to answer the following questions: - What steps will reproduce the issue? — please try to remove unnecessary steps - What's the result and what did you expect? — e.g. you can post screenshots - What error messages did you see? -- What's the affected PDF file and which pages are causing the issue? Create a pull request on [this repo](https://github.com/coolwanglu/pdf2htmlEX-testcase). +- What's the affected PDF file and which pages are causing the issue? Create a pull request on [this repo](https://github.com/pdf2htmlEX/pdf2htmlEX-testcase). Especially for issues regarding building pdf2htmlEX: - Which compiler are you using? diff --git a/INSTALL b/INSTALL index 64e4613..574e0a6 100644 --- a/INSTALL +++ b/INSTALL @@ -1,3 +1,3 @@ For instructions of building the source code, visit: -https://github.com/coolwanglu/pdf2htmlEX/wiki/Building +https://github.com/pdf2htmlEX/pdf2htmlEX/wiki/Building diff --git a/LICENSE b/LICENSE index 90c48ff..1a196de 100644 --- a/LICENSE +++ b/LICENSE @@ -1,4 +1,4 @@ -pdf2htmlEX (https://github.com/coolwanglu/pdf2htmlEX) +pdf2htmlEX (https://github.com/pdf2htmlEX/pdf2htmlEX) Copyright (c) 2012-2014 Lu Wang and other contributors pdf2htmlEX, as a whole package, is licensed under GPLv3 (or any later diff --git a/debian/changelog b/archive/debian/changelog similarity index 100% rename from debian/changelog rename to archive/debian/changelog diff --git a/debian/compat b/archive/debian/compat similarity index 100% rename from debian/compat rename to archive/debian/compat diff --git a/debian/control b/archive/debian/control similarity index 100% rename from debian/control rename to archive/debian/control diff --git a/debian/copyright b/archive/debian/copyright similarity index 100% rename from debian/copyright rename to archive/debian/copyright diff --git a/debian/dirs b/archive/debian/dirs similarity index 100% rename from debian/dirs rename to archive/debian/dirs diff --git a/debian/pdf2htmlex.NEWS b/archive/debian/pdf2htmlex.NEWS similarity index 100% rename from debian/pdf2htmlex.NEWS rename to archive/debian/pdf2htmlex.NEWS diff --git a/debian/pdf2htmlex.README b/archive/debian/pdf2htmlex.README similarity index 100% rename from debian/pdf2htmlex.README rename to archive/debian/pdf2htmlex.README diff --git a/debian/pdf2htmlex.TODO b/archive/debian/pdf2htmlex.TODO similarity index 100% rename from debian/pdf2htmlex.TODO rename to archive/debian/pdf2htmlex.TODO diff --git a/debian/rules b/archive/debian/rules similarity index 100% rename from debian/rules rename to archive/debian/rules diff --git a/debian/source/format b/archive/debian/source/format similarity index 100% rename from debian/source/format rename to archive/debian/source/format diff --git a/buildScripts/Readme.md b/buildScripts/Readme.md index 144c71d..33fae62 100644 --- a/buildScripts/Readme.md +++ b/buildScripts/Readme.md @@ -35,16 +35,6 @@ For most users, you probably really want to simply download one of the [precompiled versions of `pdf2htmlEX`](https://github.com/pdf2htmlEX/pdf2htmlEX/releases): -- [AppImage](https://appimage.org/) : Download, make executable, and - run... - - This will work on most Linuxes, and most recent Windows 10. - -- [Docker](https://www.docker.com/) Image from the [`pdf2htmlEX` Docker - hub](https://hub.docker.com/orgs/pdf2htmlex/repositories). - - This will work on any machine with Docker installed. - - [Debian archive](https://en.wikipedia.org/wiki/Dpkg) : Download, [apt](https://en.wikipedia.org/wiki/APT_(software)) install locally, and run... @@ -55,15 +45,33 @@ For most users, you probably really want to simply download one of the Experienced users of Linux, may be able to repackage the `*.deb` we provide for use with their favourite package management tool. +- [AppImage](https://appimage.org/) : Download, make executable, and + run... + + This will work on most Linuxes, and most recent Windows 10. + + (It will not currently work on MacOS or Alpine based machines). + +- [Docker](https://www.docker.com/) Image from the [`pdf2htmlEX` Docker + hub](https://hub.docker.com/orgs/pdf2htmlex/repositories). + + This will work on any machine with Docker installed. + + (Note: that *advanced* use of `pdf2htmlEX` requires careful attention to + the configuration of various tools, such as fontconfig, iconv and your + locally available fonts use by the poppler and fontforge libraries. The + docker images created by the pdf2htmlEX team might not be as well + configured for *your needs* as a docker created and configured by you) + ### Building yourself -To build `pdf2htmlEX` on your own machine, inside the root directory of -a fresh clone of the +To build `pdf2htmlEX` on a Debian/Apt related machine, inside the root +directory of a fresh clone of the [pdf2htmlEX/pdf2htmlEX](https://github.com/pdf2htmlEX/pdf2htmlEX) repository, type: ``` - ./buildScripts/buildInstallLocally + ./buildScripts/buildInstallLocallyApt ``` This will automatically install all required development tools and @@ -75,6 +83,10 @@ installing `pdf2htmlEX` into /usr/local/bin. [Debian](https://www.debian.org/) based distribution. such as [Ubuntu](https://ubuntu.com/), [Linux Mint](https://linuxmint.com/), etc. +**NOTE:** there is currently an *experimental* build script, +`./buildScripts/buildInstallLocallyAlpine`, for builds in Alpine +environments. + ## The problem To provide its full functionality, the `pdf2htmlEX` sources make direct @@ -126,7 +138,7 @@ Each script can be used individually to re-run a particular step if needed. Typically, most users, will run one of the following "top-level" scripts: -1. **`buildInstallLocally`** +1. **`buildInstallLocallyApt`** (**`buildInstallLocallyAlpine`**) This will automate: @@ -138,10 +150,16 @@ Typically, most users, will run one of the following "top-level" scripts: 3. compile and install `pdf2htmlEX`. -2. **`createImages`** + The `*Apt` script will build on any machine which uses the + `apt`/`apt-get` command. - Following a successful `buildInstallLocally`, this will create the - following images: + The `*Alpine` script will build on any machine which uses the + `apk` command (Alpine). + +2. **`createImagesApt`** (**`createImagesAlpine`**) + + Following a successful `buildInstallLocallyApt`, the `createImagesApt` + shell script will create the following images: 1. AppImage @@ -149,10 +167,18 @@ Typically, most users, will run one of the following "top-level" scripts: 3. Debian archive + Following a successful `buildInstallLocallyAlpine`, the + `createImagesAlpine` shell script will create the following images: + + 1. Alpine tar file + + 2. Docker image + 3. **`runTests`** - Following a successful `buildInstallLocally`, this will run the various - 'local' tests reporting errors as they occur. + Following a successful `buildInstallLocallyApt` (or + `buildInstallLocallyAlpine` ), the `runTests` shell script will run the + various 'local' tests reporting errors as they occur. When run in [Travis-ci](https://travis-ci.org/), failing browser tests will *not* fail the overall Travis build, but will instead upload the @@ -192,6 +218,10 @@ Typically, most users, will run one of the following "top-level" scripts: ensures that any versions of Poppler already installed by the user, are not broken by the user's installation of `pdf2htmlEX`. +- **`createAlpineTarFile`**: Using an already compiled version of `pdf2htmlEX`, + installs it and `popper-data` into a tar file suitable for use in any + Alpine environment. + - **`createAppImage`**: Using an already compiled version of `pdf2htmlEX`, installs it and `popper-data` into an AppImage. @@ -199,15 +229,28 @@ Typically, most users, will run one of the following "top-level" scripts: `pdf2htmlEX`, installs it and `poppler-data` into a Debian archive (`*.deb`). +- **`createDockerAlpineImageFromTarFile`**: Installs the Alpine tar file + archive of `pdf2htmlEX` created by `createAlpineTarFile` into an Alpine + Docker image. + - **`createDockerUbuntuImageFromDeb`**: Installs the Debian archive of `pdf2htmlEX` created by `createDebianPackage` into a Docker image. -- **`getBuildTools`**: Locally `apt` installs all development *tools* +- **`getBuildToolsAlpine`**: Locally `apk` installs all development + *tools* required to build `pdf2htmlEX`. + +- **`getBuildToolsApt`**: Locally `apt` installs all development *tools* required to build `pdf2htmlEX`. -- **`getDevLibraries`**: Locally `apt` installs all development +- **`getDevLibrariesAlpine`**: Locally `apk` installs all development *libraries* required to build `pdf2htmlEX`. +- **`getDevLibrariesApt`**: Locally `apt` installs all development + *libraries* required to build `pdf2htmlEX`. + + This script provides a definitive list of all libraries required to run + `pdf2htmlEX`. + This script provides a definitive list of all libraries required to run `pdf2htmlEX`. @@ -236,7 +279,7 @@ Typically, most users, will run one of the following "top-level" scripts: The `PDF2HTMLEX_PREFIX` variable is specified in the `versionEnvs` script. -- **`rutTests`**: Runs the tests located in the +- **`runTests`**: Runs the tests located in the `pdf2htmlEX/pdf2htmlEX/test` directory. See the `pdf2htmlEX/pdf2htmlEx/test` directory's Readme file for details. @@ -279,10 +322,6 @@ Typically, most users, will run one of the following "top-level" scripts: used by the `uploadGitHubRelease` script as the contents of the release message, as visible to the user, for the 'continuous' release section. -- **`dockerFunctions`**: A collection of shell functions used by the - `uploadDockerImage` script to automate the upload of the Docker images - to Docker Hub. - - **`listFilesByChangeTime`**: A simple shell script which lists the files in the buildScripts directory by most recently changed files first. diff --git a/buildScripts/buildInstallLocallyAlpine b/buildScripts/buildInstallLocallyAlpine index 8fe6d2f..5a535bc 100755 --- a/buildScripts/buildInstallLocallyAlpine +++ b/buildScripts/buildInstallLocallyAlpine @@ -47,11 +47,3 @@ set -ev ./buildScripts/buildPdf2htmlEX ./buildScripts/installPdf2htmlEX - -#./buildScripts/createAppImage - -#./buildScripts/createDockerUbuntuImageFromDeb - -#./buildScripts/uploadGitHubRelease - -#./buildScripts/uploadDockerImage diff --git a/buildScripts/buildInstallLocallyApt b/buildScripts/buildInstallLocallyApt index 2dcded2..a26afc4 100755 --- a/buildScripts/buildInstallLocallyApt +++ b/buildScripts/buildInstallLocallyApt @@ -47,11 +47,3 @@ set -ev ./buildScripts/buildPdf2htmlEX ./buildScripts/installPdf2htmlEX - -#./buildScripts/createAppImage - -#./buildScripts/createDockerUbuntuImageFromDeb - -#./buildScripts/uploadGitHubRelease - -#./buildScripts/uploadDockerImage diff --git a/buildScripts/createAlpineTarFile b/buildScripts/createAlpineTarFile new file mode 100755 index 0000000..5d99e56 --- /dev/null +++ b/buildScripts/createAlpineTarFile @@ -0,0 +1,79 @@ +#!/bin/sh + +# This shell script creates a docker image from an existing pdf2htmlEX +# +# This is the part which can run *inside* a docker container. + +echo "" +echo "-------------------------------------------------------------------" +echo "CREATING pdf2htmlEX Docker Image (run inside a docker container)" +echo " (based on Alpine linux $BUILD_DIST)" +echo "-------------------------------------------------------------------" +echo "" + +# Collect everything that will be needed... + +# source buildScripts/reSourceVersionEnvs +. buildScripts/reSourceVersionEnvs + +export ALPINE_NAME="pdf2htmlEX-$PDF2HTMLEX_NAME" + +echo "export ALPINE_NAME=\"$ALPINE_NAME\"" >> buildScripts/reSourceVersionEnvs + +set -ev + +mkdir -p imageBuild/alpineTarDir + +cd pdf2htmlEX/build + +sudo rm -rf install_manifest.txt + +make install DESTDIR=../../imageBuild/alpineTarDir + +cd ../../poppler-data + +make install \ + prefix=$PDF2HTMLEX_PREFIX \ + datadir=$PDF2HTMLEX_PREFIX/share/pdf2htmlEX \ + DESTDIR=../imageBuild/alpineTarDir + +cd ../imageBuild + +tar czvf $ALPINE_TARFILE_NAME.tar.gz alpineTarDir + +cat < $ALPINE_NAME.install +#!/bin/sh + +# This (alpine) shell script installs a locally existing tar.gz file of +# the pdf2htmlEX binaries compiled on Alpine $DIST. +# +# You MUST have root/sudo privileges to run this file. +# + +# We start by installing all of the (known) required runtime dependencies +# +apk update +# +apk add --no-cache \ + tar \ + libstdc++ \ + libgcc \ + gnu-libiconv \ + gettext \ + glib \ + freetype \ + fontconfig \ + cairo \ + libpng \ + libjpeg-turbo \ + libxml2 + +# Now we install the (Alpine $DIST) compiled pdf2htmlEX binaries and +# configuration files. +# +cd / +# +tar xvf $ALPINE_NAME.tar.gz --strip-comonents=1 + +ALPINE_INSTALL + diff --git a/buildScripts/createDockerAlpineImageOutside b/buildScripts/createDockerAlpineImageFromTarFile similarity index 81% rename from buildScripts/createDockerAlpineImageOutside rename to buildScripts/createDockerAlpineImageFromTarFile index ff6fa44..2ff5954 100755 --- a/buildScripts/createDockerAlpineImageOutside +++ b/buildScripts/createDockerAlpineImageFromTarFile @@ -16,15 +16,14 @@ echo "" # source buildScripts/reSourceVersionEnvs . buildScripts/reSourceVersionEnvs -# source buildScripts/dockerFunctions -. buildScripts/dockerFunctions - set -ev mkdir -p imageBuild/dockerDir cd imageBuild/dockerDir +cp ../$ALPINE_NAME.* . + if [ -z "$DOCKER_FROM" ]; then echo "" read -p "Enter the docker image for the 'from' base: " DOCKER_FROM @@ -54,21 +53,9 @@ echo "export DOCKER_NAME=\"$DOCKER_NAME\"" >> ../../buildScripts/reSourc cat > Dockerfile <> ../../buildScripts/reSourc cat > Dockerfile <" | awk '{print $3}' \ - | xargs -I '{}' echo '{}' >> DEPSFILE - done - - DEPS=$(cat DEPSFILE | sort | uniq) - - for FILE in $DEPS ; do - if [ -e $FILE ] && [[ $(readlink -f $FILE)/ != $PWD/* ]] ; then - cp -v --parents -rfL $FILE ./ || true - fi - done - rm -f DEPSFILE -} - -# Delete blacklisted files. Our blacklisted files are the shared libraries -# extracted from the "base" docker image -# -delete_blacklisted() -{ - docker run --rm -it --entrypoint=find $DOCKER_FROM -iname "*.so.*" \ - | xargs -i basename {} | sed 's/\r//g' | sed 's/\(so\.[0-9]\+\).*$/\1/' \ - | sort | uniq > BLACKLISTED_FILES - - for FILE in $(cat BLACKLISTED_FILES | tr "\r\n" " ") ; do - FILES="$(find . -name "${FILE}*" -not -path "./usr/optional/*")" - for FOUND in $FILES ; do - rm -vf "$FOUND" "$(readlink -f "$FOUND")" - done - done - - rm BLACKLISTED_FILES - - # Do not bundle developer stuff - rm -rf usr/include || true - rm -rf usr/lib/cmake || true - rm -rf usr/lib/pkgconfig || true - find . -name '*.la' | xargs -i rm {} -} - diff --git a/pdf2htmlEX/pdf2htmlEX.1.in b/pdf2htmlEX/pdf2htmlEX.1.in index d3cd881..570997c 100644 --- a/pdf2htmlEX/pdf2htmlEX.1.in +++ b/pdf2htmlEX/pdf2htmlEX.1.in @@ -359,7 +359,7 @@ pdf2htmlEX is written by Lu Wang .SH SEE ALSO .TP Home page -https://github.com/coolwanglu/pdf2htmlEX +https://github.com/pdf2htmlEX/pdf2htmlEX .TP pdf2htmlEX Wiki -https://github.com/coolwanglu/pdf2htmlEX/wiki +https://github.com/pdf2htmlEX/pdf2htmlEX/wiki diff --git a/pdf2htmlEX/share/base.css.in b/pdf2htmlEX/share/base.css.in index 515134d..7b2feac 100644 --- a/pdf2htmlEX/share/base.css.in +++ b/pdf2htmlEX/share/base.css.in @@ -2,7 +2,7 @@ /*! * Base CSS for pdf2htmlEX * Copyright 2012,2013 Lu Wang - * https://github.com/coolwanglu/pdf2htmlEX/blob/master/share/LICENSE + * https://github.com/pdf2htmlEX/pdf2htmlEX/blob/master/share/LICENSE */ /* Part 1: Web Page Layout: Free to modify, except for a few of them which are required by pdf2htmlEX.js, see the comments */ #sidebar { /* Sidebar */ diff --git a/pdf2htmlEX/share/fancy.css.in b/pdf2htmlEX/share/fancy.css.in index bdba3f6..138b6f2 100644 --- a/pdf2htmlEX/share/fancy.css.in +++ b/pdf2htmlEX/share/fancy.css.in @@ -2,7 +2,7 @@ /*! * Fancy styles for pdf2htmlEX * Copyright 2012,2013 Lu Wang - * https://github.com/coolwanglu/pdf2htmlEX/blob/master/share/LICENSE + * https://github.com/pdf2htmlEX/pdf2htmlEX/blob/master/share/LICENSE */ @keyframes fadein { from { opacity:0;} to { opacity:1;} } @-webkit-keyframes fadein { from { opacity:0;} to { opacity:1;} } diff --git a/pdf2htmlEX/share/manifest b/pdf2htmlEX/share/manifest index b03560f..0380867 100644 --- a/pdf2htmlEX/share/manifest +++ b/pdf2htmlEX/share/manifest @@ -17,7 +17,7 @@ # Declaration - Do not modify """ - + diff --git a/pdf2htmlEX/share/pdf2htmlEX.js.in b/pdf2htmlEX/share/pdf2htmlEX.js.in index a678759..9423ce9 100644 --- a/pdf2htmlEX/share/pdf2htmlEX.js.in +++ b/pdf2htmlEX/share/pdf2htmlEX.js.in @@ -2,7 +2,7 @@ /** * @license pdf2htmlEX.js: Core UI functions for pdf2htmlEX * Copyright 2012,2013 Lu Wang and other contributors - * https://github.com/coolwanglu/pdf2htmlEX/blob/master/share/LICENSE + * https://github.com/pdf2htmlEX/pdf2htmlEX/blob/master/share/LICENSE */ /* diff --git a/pdf2htmlEX/test/test_local_browser.py b/pdf2htmlEX/test/test_local_browser.py index 7c67a3d..62a3657 100755 --- a/pdf2htmlEX/test/test_local_browser.py +++ b/pdf2htmlEX/test/test_local_browser.py @@ -43,4 +43,7 @@ class test_local_browser(BrowserTests, unittest.TestCase): self.browser.save_screenshot(png_file) if __name__ == '__main__': - unittest.main() + # turn warnings OFF to prevent 'ResourceWarning: unclosed Date: Sun, 21 Jun 2020 14:02:44 +0100 Subject: [PATCH 69/69] corrected alpine image creation --- buildScripts/createAlpineTarFile | 3 ++- buildScripts/createDockerAlpineImageFromTarFile | 5 +++-- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/buildScripts/createAlpineTarFile b/buildScripts/createAlpineTarFile index 5d99e56..8022180 100755 --- a/buildScripts/createAlpineTarFile +++ b/buildScripts/createAlpineTarFile @@ -39,7 +39,7 @@ make install \ cd ../imageBuild -tar czvf $ALPINE_TARFILE_NAME.tar.gz alpineTarDir +tar czvf $ALPINE_NAME.tar.gz alpineTarDir cat < $ALPINE_NAME.install #!/bin/sh @@ -77,3 +77,4 @@ tar xvf $ALPINE_NAME.tar.gz --strip-comonents=1 ALPINE_INSTALL +chmod a+x $ALPINE_NAME.tar.gz diff --git a/buildScripts/createDockerAlpineImageFromTarFile b/buildScripts/createDockerAlpineImageFromTarFile index 2ff5954..5f1edf4 100755 --- a/buildScripts/createDockerAlpineImageFromTarFile +++ b/buildScripts/createDockerAlpineImageFromTarFile @@ -53,9 +53,10 @@ echo "export DOCKER_NAME=\"$DOCKER_NAME\"" >> ../../buildScripts/reSourc cat > Dockerfile <