diff --git a/.travis.yml b/.travis.yml index b2a16e5..24e0ff6 100644 --- a/.travis.yml +++ b/.travis.yml @@ -17,26 +17,12 @@ branches: jobs: fast_finish: true include: - - stage: build-linux - os: linux -# env: -# global: DOCKER_FROM="ubuntu:18.04" -# services: -# - docker + - os: linux dist: bionic + env: DOCKER_FROM="ubuntu:18.04" + services: + - docker + addons: + firefox: latest script: - ./buildScripts/travisLinuxDoItAll -# - stage: test -# os: -# - linux -# script: ./testScripts/runLocalTests -#deploy: -# provider: releases -# api_key: -# secure: ALWKAkgeLSPS4ULyTtxU50ZmSICEafCirCQ1YmuVTqe9nrc8E9Jm+4QaBlab5k6v90EpzNvZBnMOiT8cT1Wp2lqJzcGdGMcG49mhRvirdYhXgWuHYnygVgkxEhgejXcQaEiaGPDV3Y1k9MuiAVr8ApjUvIMlY4Hg1y0r7/rqR2fYmen+f+6a87bp5PptwakC9bfoYb92651d0TbugFfKT6TGKlE2jGlQ6On1alzLQ2DXA8coUjwpp4NcDD8OhnYGN/VZWkqckaBrMrHwjJ5FTfy5MGBOqCLzNix8sCGLEeCKkKC8J0nniNJ7+KOu9LJ2uIezKYfaUOJV0kik7F9ErvxjbfSRieako2yaSOR8X39aicTtpbyoXPRSKDU4WYX56vt4Dx7aNrYtP5b/UwWGXZa9MK7rNGJ+hiMDO1AzuyXirzR4RaJXCyC8n5W9ahBZBHCMqd5gMg5sMGPNj3Rgt34+R+nGYssum5wvr+6IKNl57sIoZFCgqviBrw2gJOCqlgOcEk/4ZSehWnXPCk6Gzxvkqkt/dmp8pJLNyvMaWcEZ6CXNHDU8MZS7bASo8Z7eAYvmX0MMjVty8J47expVaJ8M3OFHu6lE6tU8wwfd+Igr2TlWcixqaa3hDMgS+ghwdiZbYuMdR1AyZnEqK3xwvXXPPXrqRsB9xHtczLibkjY= -# file: pdf2htmlEX-newBuildSystem-x86_64.AppImage -# skip_cleanup: true -# on: -# tags: true -# repo: stephengaito/pdf2htmlEX - diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 72508e2..51159c6 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -23,7 +23,7 @@ A few channels are available to reach the developers, please find the most prope ### The Issue Tracker -The [Issue Tracker](https://github.com/coolwanglu/pdf2htmlEX/issues) +The [Issue Tracker](https://github.com/pdf2htmlEX/pdf2htmlEX/issues) is the best way for [bug reports](#bug-reports), [features requests](#feature-requests) @@ -54,7 +54,7 @@ However do not report issues or submit patches there, since it's terrible to kee pdf2htmlEX is mostly written and maintained by 王璐 (Lu Wang). His email and twitter account can be found in -[README.md](https://github.com/coolwanglu/pdf2htmlEX/blob/master/README.md). +[README.md](https://github.com/pdf2htmlEX/pdf2htmlEX/blob/master/README.md). Please post only messages that do not fit into the above channels, otherwise note that he no longer replies with _Please post your question to the mailing list_ or _Please file an issue at GitHub_, consider your message already replied. @@ -73,7 +73,7 @@ follow the following steps to get it resolved as fast as possible: First of all, did you realize that your question might have been already answered in one of the following places? -- [pdf2htmlEX Wiki](https://github.com/coolwanglu/pdf2htmlEX/wiki) +- [pdf2htmlEX Wiki](https://github.com/pdf2htmlEX/pdf2htmlEX/wiki) - The manpage (run `man pdf2htmlEX`) - Old posts in the [mailing list](#the-mailing-list) or the [issue tracker](#the-issue-tracker) - [Google](http://www.google.com/) @@ -93,7 +93,7 @@ Bugs should always be reported to [the Issue Tracker](#the-issue-tracker). Before you report any bug: - Use the latest git version of pdf2htmlEX, since the issue may have been already fixed. - Search for previous issues (open or closed), to make sure that the issue has not been reported before. -- If pdf2htmlEX crashed, take a look at [this article](https://github.com/coolwanglu/pdf2htmlEX/wiki/Troubleshooting-Crashes). +- If pdf2htmlEX crashed, take a look at [this article](https://github.com/pdf2htmlEX/pdf2htmlEX/wiki/Troubleshooting-Crashes). A good bug report shouldn't leave others needing to chase you up for more information. The developers may be very familiar with the code base of pdf2htmlEX, @@ -110,7 +110,7 @@ If you are not sure, please try to answer the following questions: - What steps will reproduce the issue? — please try to remove unnecessary steps - What's the result and what did you expect? — e.g. you can post screenshots - What error messages did you see? -- What's the affected PDF file and which pages are causing the issue? Create a pull request on [this repo](https://github.com/coolwanglu/pdf2htmlEX-testcase). +- What's the affected PDF file and which pages are causing the issue? Create a pull request on [this repo](https://github.com/pdf2htmlEX/pdf2htmlEX-testcase). Especially for issues regarding building pdf2htmlEX: - Which compiler are you using? diff --git a/INSTALL b/INSTALL index 64e4613..574e0a6 100644 --- a/INSTALL +++ b/INSTALL @@ -1,3 +1,3 @@ For instructions of building the source code, visit: -https://github.com/coolwanglu/pdf2htmlEX/wiki/Building +https://github.com/pdf2htmlEX/pdf2htmlEX/wiki/Building diff --git a/LICENSE b/LICENSE index 90c48ff..1a196de 100644 --- a/LICENSE +++ b/LICENSE @@ -1,4 +1,4 @@ -pdf2htmlEX (https://github.com/coolwanglu/pdf2htmlEX) +pdf2htmlEX (https://github.com/pdf2htmlEX/pdf2htmlEX) Copyright (c) 2012-2014 Lu Wang and other contributors pdf2htmlEX, as a whole package, is licensed under GPLv3 (or any later diff --git a/debian/changelog b/archive/debian/changelog similarity index 100% rename from debian/changelog rename to archive/debian/changelog diff --git a/debian/compat b/archive/debian/compat similarity index 100% rename from debian/compat rename to archive/debian/compat diff --git a/debian/control b/archive/debian/control similarity index 100% rename from debian/control rename to archive/debian/control diff --git a/debian/copyright b/archive/debian/copyright similarity index 100% rename from debian/copyright rename to archive/debian/copyright diff --git a/debian/dirs b/archive/debian/dirs similarity index 100% rename from debian/dirs rename to archive/debian/dirs diff --git a/debian/pdf2htmlex.NEWS b/archive/debian/pdf2htmlex.NEWS similarity index 100% rename from debian/pdf2htmlex.NEWS rename to archive/debian/pdf2htmlex.NEWS diff --git a/debian/pdf2htmlex.README b/archive/debian/pdf2htmlex.README similarity index 100% rename from debian/pdf2htmlex.README rename to archive/debian/pdf2htmlex.README diff --git a/debian/pdf2htmlex.TODO b/archive/debian/pdf2htmlex.TODO similarity index 100% rename from debian/pdf2htmlex.TODO rename to archive/debian/pdf2htmlex.TODO diff --git a/debian/rules b/archive/debian/rules similarity index 100% rename from debian/rules rename to archive/debian/rules diff --git a/debian/source/format b/archive/debian/source/format similarity index 100% rename from debian/source/format rename to archive/debian/source/format diff --git a/buildScripts/Readme.md b/buildScripts/Readme.md new file mode 100644 index 0000000..33fae62 --- /dev/null +++ b/buildScripts/Readme.md @@ -0,0 +1,334 @@ +# Building pdf2htmlEX + +Because of its intimate use of *specific* versions of both Poppler and +FontForge, cleanly building `pdf2htmlEX` is rather more complex than +normal. + +The (shell) scripts in this directory help automate this mutli-stage +process. + +For all but the most experienced programmers, we *strongly* encourage you +to use these scripts to build `pdf2htmlEX`. + +--- + +**Table of contents** + +- [TL;DR ...](#tldr-) + - [Downloading precompiled versions](#downloading-precompiled-versions-downloads) + - [Building yourself](#building-yourself) +- [The problem](#the-problem) +- [Our solution](#our-solution) +- [The gory details ...](#the-gory-details-) + - [Top-level scripts](#top-level-scripts) + - [Individual steps](#individual-steps) + - [Helper files and scripts](#helper-files-and-scripts) +- [Yet more details?](#yet-more-details) + +--- + +## TL;DR ... + +### Downloading precompiled versions + +For most users, you probably really want to simply download one of the +[precompiled versions of +`pdf2htmlEX`](https://github.com/pdf2htmlEX/pdf2htmlEX/releases): + +- [Debian archive](https://en.wikipedia.org/wiki/Dpkg) : Download, + [apt](https://en.wikipedia.org/wiki/APT_(software)) install locally, + and run... + + This will work on any [Debian](https://www.debian.org/) based and most + recent Windows 10 machines. + + Experienced users of Linux, may be able to repackage the `*.deb` we + provide for use with their favourite package management tool. + +- [AppImage](https://appimage.org/) : Download, make executable, and + run... + + This will work on most Linuxes, and most recent Windows 10. + + (It will not currently work on MacOS or Alpine based machines). + +- [Docker](https://www.docker.com/) Image from the [`pdf2htmlEX` Docker + hub](https://hub.docker.com/orgs/pdf2htmlex/repositories). + + This will work on any machine with Docker installed. + + (Note: that *advanced* use of `pdf2htmlEX` requires careful attention to + the configuration of various tools, such as fontconfig, iconv and your + locally available fonts use by the poppler and fontforge libraries. The + docker images created by the pdf2htmlEX team might not be as well + configured for *your needs* as a docker created and configured by you) + +### Building yourself + +To build `pdf2htmlEX` on a Debian/Apt related machine, inside the root +directory of a fresh clone of the +[pdf2htmlEX/pdf2htmlEX](https://github.com/pdf2htmlEX/pdf2htmlEX) +repository, type: + +``` + ./buildScripts/buildInstallLocallyApt +``` + +This will automatically install all required development tools and +libraries, and then proceed to download and statically compile the +required versions of both Poppler and FontForge before compiling and +installing `pdf2htmlEX` into /usr/local/bin. + +**NOTE:** at the moment this will **only** work on machines with a +[Debian](https://www.debian.org/) based distribution. such as +[Ubuntu](https://ubuntu.com/), [Linux Mint](https://linuxmint.com/), etc. + +**NOTE:** there is currently an *experimental* build script, +`./buildScripts/buildInstallLocallyAlpine`, for builds in Alpine +environments. + +## The problem + +To provide its full functionality, the `pdf2htmlEX` sources make direct +use of source code and unexposed methods from both the Poppler and +FontForge projects. Unfortunately the source code in the Poppler and +FontForge projects that the `pdf2htmlEX` uses changes regularly. + +This means that the `pdf2htmlEX` souce code *must* be updated regularly to +match *specific releases* of both Poppler and FontForge. + +Unfortunately, the installed versions of both Poppler and FontForge in +most Linux distributions, lag the official releases of both of these +projects. Even worse few distributions install the same versions. + +This means that it is nearly impossible for the `pdf2htmlEX` code to +'predict' which version of Poppler or FontForge will be installed on a +given user's machine. + +## Our solution + +While we *could* keep multiple versions of the `pdf2htmlEX` source code, +each version matched to a particular distribution's installed versions of +Poppler and FontForge, this would be a logistic and testing 'nightmare'. + +Instead, when building `pdf2htmlEX`, we download specific versions of both +the Poppler and FontForge sources (usually the most recent), and then +compile *static* versions of the Poppler and FontForge libraries which are +then *statically* linked into the `pdf2htmlEX` binary. + +This means that the `pdf2htmlEX` binary is completely independent of any +locally installed versions of either Poppler or FontForge. + +However, to get the matched versions of Poppler and FontForge and then +compile them statically, *our* build process becomes much more complex +than a "simple", `configure, make, make install` cycle. + +Hence this directory has a large number of shell scripts each of which +automate one simple step in the overall our build process. + +## The gory details ... + +The shell scripts in this directory automate the download, build, install, +test and upload steps required to provide a complete build/test/release +cycle of `pdf2htmlEX`. + +Each script can be used individually to re-run a particular step if needed. + +### Top-level scripts + +Typically, most users, will run one of the following "top-level" scripts: + +1. **`buildInstallLocallyApt`** (**`buildInstallLocallyAlpine`**) + + This will automate: + + 1. the installation of all required development tools + and libraries, + + 2. download and statically compile the required versions of both + Poppler and FontForge, + + 3. compile and install `pdf2htmlEX`. + + The `*Apt` script will build on any machine which uses the + `apt`/`apt-get` command. + + The `*Alpine` script will build on any machine which uses the + `apk` command (Alpine). + +2. **`createImagesApt`** (**`createImagesAlpine`**) + + Following a successful `buildInstallLocallyApt`, the `createImagesApt` + shell script will create the following images: + + 1. AppImage + + 2. Docker image + + 3. Debian archive + + Following a successful `buildInstallLocallyAlpine`, the + `createImagesAlpine` shell script will create the following images: + + 1. Alpine tar file + + 2. Docker image + +3. **`runTests`** + + Following a successful `buildInstallLocallyApt` (or + `buildInstallLocallyAlpine` ), the `runTests` shell script will run the + various 'local' tests reporting errors as they occur. + + When run in [Travis-ci](https://travis-ci.org/), failing browser tests + will *not* fail the overall Travis build, but will instead upload the + test results to the GitHub Release page for later review. + +4. **`uploadImages`** + + Following successful `buildInstallLocally`, `createImages` and + `runTests`, this will automate the upload of the various artefacts to + the `pdf2htmlEX` releases page, and docker hub repository. + + **Note** that this step requires the user to enter passwords for each + of the respective services. *Most* users will not need (or be able) to + run this step. + +5. **`travisLinuxDoItAll`** + + This script is used by the `.travis.yml` configuration to build, test + and upload a complete `pdf2htmlEX` release cycle. It is essentially a + compendium of all of the build scripts in the correct order. + +### Individual steps + +- **`buildFontforge`**: Compiles a *static* version of `libfontforge` for + use by `pdf2htmlEX`. + + Statically linking `libfontforge` into `phd2htmlEX` ensures that any + versions of FontForge already installed by the user, are not broken by + the user's installation of `pdf2htmlEX`. + +- **`buildPdf2htmlEX`**: Compiles and links `pdf2htmlEX`. + +- **`buildPoppler`**: Compiles a *static* version of `libpoppler` and + `libpopper-glib` for use by `pdf2htmlEX`. + + Statically linking `libpoppler` and `libpoppler-glib` into `phd2htmlEX` + ensures that any versions of Poppler already installed by the user, are + not broken by the user's installation of `pdf2htmlEX`. + +- **`createAlpineTarFile`**: Using an already compiled version of `pdf2htmlEX`, + installs it and `popper-data` into a tar file suitable for use in any + Alpine environment. + +- **`createAppImage`**: Using an already compiled version of `pdf2htmlEX`, + installs it and `popper-data` into an AppImage. + +- **`createDebianPackage`**: Using an already compiled version of + `pdf2htmlEX`, installs it and `poppler-data` into a Debian archive + (`*.deb`). + +- **`createDockerAlpineImageFromTarFile`**: Installs the Alpine tar file + archive of `pdf2htmlEX` created by `createAlpineTarFile` into an Alpine + Docker image. + +- **`createDockerUbuntuImageFromDeb`**: Installs the Debian archive of + `pdf2htmlEX` created by `createDebianPackage` into a Docker image. + +- **`getBuildToolsAlpine`**: Locally `apk` installs all development + *tools* required to build `pdf2htmlEX`. + +- **`getBuildToolsApt`**: Locally `apt` installs all development *tools* + required to build `pdf2htmlEX`. + +- **`getDevLibrariesAlpine`**: Locally `apk` installs all development + *libraries* required to build `pdf2htmlEX`. + +- **`getDevLibrariesApt`**: Locally `apt` installs all development + *libraries* required to build `pdf2htmlEX`. + + This script provides a definitive list of all libraries required to run + `pdf2htmlEX`. + + This script provides a definitive list of all libraries required to run + `pdf2htmlEX`. + +- **`getFontforge`**: Downloads and unpacks the version of FontForge specified in the + `FONTFORGE_VERSION` environment variable into the + `pdf2htmlEX/fontoforge` directory. + + The `FONTFORGE_VERSION` variable is specified in the `versionEnvs` + script. + +- **`getPoppler`**: Downloads and unpacks the version of Poppler specified in the + `POPPLER_VERSION` environment variable into the `pdf2htmlEX/poppler` + directory. + + The `POPPLER_VERSION` variable is specified in the `versionEnvs` script. + + The `getPoppler` script also downloads and unpacks the most recent + version of `poppler-data`. Since `poppler-data` does not change very + often, the correct version of `poppler-data` is specified in the + `getPoppler` script itself. + +- **`installPdf2htmlEX`**: Installs an already compiled version of + `pdf2htmlEX` and `poppler-data` into the location specified by the + `PDF2HTMLEX_PREFIX` environment variable. + + The `PDF2HTMLEX_PREFIX` variable is specified in the `versionEnvs` + script. + +- **`runTests`**: Runs the tests located in the + `pdf2htmlEX/pdf2htmlEX/test` directory. See the + `pdf2htmlEX/pdf2htmlEx/test` directory's Readme file for details. + +- **`uploadDockerImage`**: Upload the `pdf2htmlEX` Docker image to + Docker hub repository associated to the docker hub users specified in + the `DOCKER_USERNAME` environement variable. + + Unless the `DOCKER_USERNAME` and `DOCKER_PASSWORD` environment variables + are pre-defined, this script will prompt the user for the respective + values. + +- **`uploadGitHubRelease`**: Upload the `pdf2htmlEX` artefacts (AppImage, + Debian archive, test results, etc) to the *continuous* section of the + release page associated with the `TRAVIS_REPO_SLUG` (user/project) + environment variable. + + Unless the `GITHUB_USERNAME`, `GITHUB_TOKEN`, and `TRAVIS_REPO_SLUG` + (user/project) environment variables are pre-defined, this script will + prompt the user for the respective values. + +### Helper files and scripts + +- **`versionEnvs`**: Specifies all of the evnironment variables required + for a standard build of `pdf2htmlEX`. Changes in this script effect + *all* of the other build scripts. + +- **`reSourceVersionEnvs`**: This shell script is automatically generated + by the build scripts as they are run. It records the values of all + important environment variables required by the buildScripts. It is + typcically `source`d by each script before it preforms any actions. + +- **`reportEnvs`**: Echos all important enviroment variables to the + console. This script is used by the top-level scripts to ensure the + current environment variables are listed before each build. + +- **`uploadGitHubReleaseDSL`**: A collection of shell functions used by the + `uploadGitHubRelease` script to automate the upload of release artefacts. + +- **`uploadGitHubReleaseMessage`**: The contents of this *text* file is + used by the `uploadGitHubRelease` script as the contents of the release + message, as visible to the user, for the 'continuous' release section. + +- **`listFilesByChangeTime`**: A simple shell script which lists the files + in the buildScripts directory by most recently changed files first. + +- **`Readme.md`**: This read me file. + +## Yet more details? + +The various shell script files are meant to be fairly readable. They +contain additional comments about what each step is meant to be doing. + diff --git a/buildScripts/buildFontforge b/buildScripts/buildFontforge index 26cf38a..e58d495 100755 --- a/buildScripts/buildFontforge +++ b/buildScripts/buildFontforge @@ -1,8 +1,9 @@ -#!/bin/bash +#!/bin/sh -# This bash script build FontForge +# This shell script build FontForge -source buildScripts/reSourceVersionEnvs +# source buildScripts/reSourceVersionEnvs +. buildScripts/reSourceVersionEnvs echo "" echo "-------------------------------------------------------------------" diff --git a/buildScripts/buildInstallLocally b/buildScripts/buildInstallLocallyAlpine similarity index 76% rename from buildScripts/buildInstallLocally rename to buildScripts/buildInstallLocallyAlpine index 11c0d69..5a535bc 100755 --- a/buildScripts/buildInstallLocally +++ b/buildScripts/buildInstallLocallyAlpine @@ -1,6 +1,6 @@ -#!/bin/bash +#!/bin/sh -# This bash script builds the complete pdf2htmlEX application LOCALLY +# This shell script builds the complete pdf2htmlEX application LOCALLY # (It does not create the AppImage or Docker images) # Adjust the following two environment variables to suit your needs @@ -32,9 +32,9 @@ set -ev ./buildScripts/reportEnvs -./buildScripts/getBuildTools +./buildScripts/getBuildToolsAlpine -./buildScripts/getDevLibraries +./buildScripts/getDevLibrariesAlpine ./buildScripts/getPoppler @@ -47,11 +47,3 @@ set -ev ./buildScripts/buildPdf2htmlEX ./buildScripts/installPdf2htmlEX - -#./buildScripts/createAppImage - -#./buildScripts/createDockerImage - -#./buildScripts/uploadGitHubRelease - -#./buildScripts/uploadDockerImage diff --git a/buildScripts/buildInstallLocallyApt b/buildScripts/buildInstallLocallyApt new file mode 100755 index 0000000..a26afc4 --- /dev/null +++ b/buildScripts/buildInstallLocallyApt @@ -0,0 +1,49 @@ +#!/bin/sh + +# This shell script builds the complete pdf2htmlEX application LOCALLY +# (It does not create the AppImage or Docker images) + +# Adjust the following two environment variables to suit your needs +# +export UNATTENDED="--assume-yes" +export MAKE_PARALLEL="-j $(nproc)" + +# choose one of the following... +# +# export PDF2HTMLEX_BRANCH="<>" +export PDF2HTMLEX_BRANCH="$(git rev-parse --abbrev-ref HEAD)" + +# The following environment variable determines where the poppler, +# poppler-data, fontforge and pdf2htmlEX packages are installed. +# CHANGE IT TO SUIT YOUR NEEDS: +# +export PDF2HTMLEX_PREFIX=/usr/local + +# Ensure all Apt packages are installed with no user interaction +# +export DEBIAN_FRONTEND=noninteractive + +set -ev + +################ +# do the build + +./buildScripts/versionEnvs + +./buildScripts/reportEnvs + +./buildScripts/getBuildToolsApt + +./buildScripts/getDevLibrariesApt + +./buildScripts/getPoppler + +./buildScripts/buildPoppler + +./buildScripts/getFontforge + +./buildScripts/buildFontforge + +./buildScripts/buildPdf2htmlEX + +./buildScripts/installPdf2htmlEX diff --git a/buildScripts/buildPdf2htmlEX b/buildScripts/buildPdf2htmlEX index 5a048bb..daa62a3 100755 --- a/buildScripts/buildPdf2htmlEX +++ b/buildScripts/buildPdf2htmlEX @@ -1,6 +1,6 @@ -#!/bin/bash +#!/bin/sh -# This bash script builds pdf2htmlEX +# This shell script builds pdf2htmlEX echo "" echo "-------------------------------------------------------------------" @@ -12,6 +12,7 @@ echo "" set -ev cd pdf2htmlEX +rm -rf build mkdir build cd build cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$PDF2HTMLEX_PREFIX .. diff --git a/buildScripts/buildPoppler b/buildScripts/buildPoppler index 3253c75..4583ccb 100755 --- a/buildScripts/buildPoppler +++ b/buildScripts/buildPoppler @@ -1,6 +1,6 @@ -#!/bin/bash +#!/bin/sh -# This bash script builds the latest poppler +# This shell script builds the latest poppler echo "" echo "-------------------------------------------------------------------" diff --git a/buildScripts/cleanPdf2htmlEX b/buildScripts/cleanPdf2htmlEX deleted file mode 100755 index a01e378..0000000 --- a/buildScripts/cleanPdf2htmlEX +++ /dev/null @@ -1,6 +0,0 @@ -#!/bin/bash - -# This bash script clean all pdf2htmlEX build products - -cd pdf2htmlEX -rm -rf build pdf2htmlEX.1 share/*.css share/*.js share/*.min.* src/pdf2htmlEX-config.h src/util/css_const.h diff --git a/buildScripts/createAlpineTarFile b/buildScripts/createAlpineTarFile new file mode 100755 index 0000000..8022180 --- /dev/null +++ b/buildScripts/createAlpineTarFile @@ -0,0 +1,80 @@ +#!/bin/sh + +# This shell script creates a docker image from an existing pdf2htmlEX +# +# This is the part which can run *inside* a docker container. + +echo "" +echo "-------------------------------------------------------------------" +echo "CREATING pdf2htmlEX Docker Image (run inside a docker container)" +echo " (based on Alpine linux $BUILD_DIST)" +echo "-------------------------------------------------------------------" +echo "" + +# Collect everything that will be needed... + +# source buildScripts/reSourceVersionEnvs +. buildScripts/reSourceVersionEnvs + +export ALPINE_NAME="pdf2htmlEX-$PDF2HTMLEX_NAME" + +echo "export ALPINE_NAME=\"$ALPINE_NAME\"" >> buildScripts/reSourceVersionEnvs + +set -ev + +mkdir -p imageBuild/alpineTarDir + +cd pdf2htmlEX/build + +sudo rm -rf install_manifest.txt + +make install DESTDIR=../../imageBuild/alpineTarDir + +cd ../../poppler-data + +make install \ + prefix=$PDF2HTMLEX_PREFIX \ + datadir=$PDF2HTMLEX_PREFIX/share/pdf2htmlEX \ + DESTDIR=../imageBuild/alpineTarDir + +cd ../imageBuild + +tar czvf $ALPINE_NAME.tar.gz alpineTarDir + +cat < $ALPINE_NAME.install +#!/bin/sh + +# This (alpine) shell script installs a locally existing tar.gz file of +# the pdf2htmlEX binaries compiled on Alpine $DIST. +# +# You MUST have root/sudo privileges to run this file. +# + +# We start by installing all of the (known) required runtime dependencies +# +apk update +# +apk add --no-cache \ + tar \ + libstdc++ \ + libgcc \ + gnu-libiconv \ + gettext \ + glib \ + freetype \ + fontconfig \ + cairo \ + libpng \ + libjpeg-turbo \ + libxml2 + +# Now we install the (Alpine $DIST) compiled pdf2htmlEX binaries and +# configuration files. +# +cd / +# +tar xvf $ALPINE_NAME.tar.gz --strip-comonents=1 + +ALPINE_INSTALL + +chmod a+x $ALPINE_NAME.tar.gz diff --git a/buildScripts/createAppImage b/buildScripts/createAppImage index 87f99f3..1a72152 100755 --- a/buildScripts/createAppImage +++ b/buildScripts/createAppImage @@ -1,8 +1,9 @@ -#!/bin/bash +#!/bin/sh -# This bash script creates an AppImage for pdf2htmlEX +# This shell script creates an AppImage for pdf2htmlEX -source ./buildScripts/reSourceVersionEnvs +# source ./buildScripts/reSourceVersionEnvs +. ./buildScripts/reSourceVersionEnvs echo "" echo "-------------------------------------------------------------------" @@ -14,8 +15,8 @@ set -ev # For appimage output plugin # -export VERSION="$PDF2HTMLEX_BRANCH-$BUILD_TIME" -export APPIMAGE_NAME="pdf2htmlEX-$PDF2HTMLEX_BRANCH-$BUILD_TIME-$MACHINE_ARCH.AppImage" +#export VERSION="$PDF2HTMLEX_BRANCH-$BUILD_TIME" +export APPIMAGE_NAME="pdf2htmlEX-$PDF2HTMLEX_NAME.AppImage" echo "export APPIMAGE_NAME=\"$APPIMAGE_NAME\"" >> buildScripts/reSourceVersionEnvs @@ -36,6 +37,14 @@ make install \ cd ../imageBuild +# Make sure directories can be traversed by nobody +# +#find appDir -type d -exec chmod 755 {} \; +# +# Make sure files can be read by nobody +# +#find appDir -type f -exec chmod 644 {} \; + # force libfontconfig into AppImage (linuxdeploy blacklists libfontconfig) # (turned off since libfontconfig needs to be matched to the underlying # OS's collection of fonts and /etc/fonts configuration files) @@ -59,6 +68,9 @@ if [ -f /.dockerenv ] ; then LINUX_DEPLOY_APP_IMAGE=squashfs-root/AppRun fi +export VERBOSE=1 +export OUTPUT=$APPIMAGE_NAME + ./$LINUX_DEPLOY_APP_IMAGE \ -e appDir/$PDF2HTMLEX_PREFIX/bin/pdf2htmlEX \ --create-desktop-file \ diff --git a/buildScripts/createDebianPackage b/buildScripts/createDebianPackage index 47c9d8d..3c47d5f 100755 --- a/buildScripts/createDebianPackage +++ b/buildScripts/createDebianPackage @@ -1,8 +1,9 @@ -#!/bin/bash +#!/bin/sh -# This bash script creates a (binary) Debian Package Archive for pdf2htmlEX +# This shell script creates a (binary) Debian Package Archive for pdf2htmlEX -source ./buildScripts/reSourceVersionEnvs +# source ./buildScripts/reSourceVersionEnvs +. ./buildScripts/reSourceVersionEnvs echo "" echo "-------------------------------------------------------------------" @@ -10,11 +11,9 @@ echo "CREATING pdf2htmlEX (binary) Debian package" echo "-------------------------------------------------------------------" echo "" -source /etc/lsb-release - set -ev -export DPKG_NAME="pdf2htmlEX-$PDF2HTMLEX_BRANCH-$BUILD_TIME-$MACHINE_ARCH-$DISTRIB_CODENAME.deb" +export DPKG_NAME="pdf2htmlEX-$PDF2HTMLEX_NAME.deb" echo "export DPKG_NAME=\"$DPKG_NAME\"" >> buildScripts/reSourceVersionEnvs @@ -69,7 +68,7 @@ find $DEBDIR -type f | xargs md5sum > $md5sumsFile # Accumulate the control file information # -versionValue=$(git describe --abbrev=0) +versionValue=$PDF2HTMLEX_VERSION.$PDF2HTMLEX_BRANCH.$BUILD_DIST.$BUILD_DATE architectureValue=$(dpkg-architecture -q DEB_BUILD_ARCH_CPU) maintainerValue="$(git config --get user.name) <$(git config --get user.email)>" @@ -77,7 +76,7 @@ maintainerValue="$(git config --get user.name) <$(git config --get user.email)>" # echo "Package: pdf2htmlEX" > $controlFile echo "Version: 0:0.$versionValue-0" >> $controlFile -echo "Distribution: $DISTRIB_CODENAME" >> $controlFile +echo "Distribution: $BUILD_DIST" >> $controlFile echo "Architecture: $architectureValue" >> $controlFile echo "Section: universe/web" >> $controlFile echo "Priority: optional" >> $controlFile @@ -96,6 +95,18 @@ touch $conffilesFile # cd imageBuild # +# Make sure directories can be traversed by nobody +# +#find debianDir -type d -exec chmod 755 {} \; +# +# Make sure files can be read by nobody +# +#find debianDir -type f -exec chmod 644 {} \; +# +# Make sure root:root owns all files +# sudo chown -R root:root debianDir # +# Build the package +# dpkg --build debianDir $DPKG_NAME diff --git a/buildScripts/createDockerAlpineImageFromTarFile b/buildScripts/createDockerAlpineImageFromTarFile new file mode 100755 index 0000000..5f1edf4 --- /dev/null +++ b/buildScripts/createDockerAlpineImageFromTarFile @@ -0,0 +1,66 @@ +#!/bin/sh + +# This shell script creates a docker image from an existing pdf2htmlEX +# +# This is the part which must be run *outside* of any docker container. + +echo "" +echo "-------------------------------------------------------------------" +echo "CREATING pdf2htmlEX Docker Image (run outside any docker container)" +echo " (based on Alpine linux $BUILD_DIST)" +echo "-------------------------------------------------------------------" +echo "" + +# Collect everything that will be needed... + +# source buildScripts/reSourceVersionEnvs +. buildScripts/reSourceVersionEnvs + +set -ev + +mkdir -p imageBuild/dockerDir + +cd imageBuild/dockerDir + +cp ../$ALPINE_NAME.* . + +if [ -z "$DOCKER_FROM" ]; then + echo "" + read -p "Enter the docker image for the 'from' base: " DOCKER_FROM + echo "" + if [ -z "$DOCKER_FROM" ]; then + echo "DOCKER_FROM not set... so we can not build the docker image" + exit 1 + fi +fi + +if [ -z "$DOCKER_USERNAME" ]; then + echo "" + read -p "Enter a docker username: " DOCKER_USERNAME + echo "" + if [ -z "$DOCKER_USERNAME" ]; then + echo "DOCKER_USERNAME not set... so we can not build the docker image" + exit 1 + fi +fi + +export DOCKER_NAME="$DOCKER_USERNAME/pdf2htmlex:$PDF2HTMLEX_NAME" + +echo "export DOCKER_FROM=\"$DOCKER_FROM\"" >> ../../buildScripts/reSourceVersionEnvs +echo "export DOCKER_USERNAME=\"$DOCKER_USERNAME\"" >> ../../buildScripts/reSourceVersionEnvs +echo "export DOCKER_NAME=\"$DOCKER_NAME\"" >> ../../buildScripts/reSourceVersionEnvs + +cat > Dockerfile <> buildScripts/reSourceVersionEnvs - echo "export DOCKER_USERNAME=\"$DOCKER_USERNAME\"" >> buildScripts/reSourceVersionEnvs - echo "export DOCKER_NAME=\"$DOCKER_NAME\"" >> buildScripts/reSourceVersionEnvs - - delete_blacklisted - - cat > Dockerfile <> ../../buildScripts/reSourceVersionEnvs +echo "export DOCKER_USERNAME=\"$DOCKER_USERNAME\"" >> ../../buildScripts/reSourceVersionEnvs +echo "export DOCKER_NAME=\"$DOCKER_NAME\"" >> ../../buildScripts/reSourceVersionEnvs + +cat > Dockerfile <" | awk '{print $3}' \ - | xargs -I '{}' echo '{}' >> DEPSFILE - done - - DEPS=$(cat DEPSFILE | sort | uniq) - - for FILE in $DEPS ; do - if [ -e $FILE ] && [[ $(readlink -f $FILE)/ != $PWD/* ]] ; then - cp -v --parents -rfL $FILE ./ || true - fi - done - rm -f DEPSFILE -} - -# Delete blacklisted files. Our blacklisted files are the shared libraries -# extracted from the "base" docker image -# -delete_blacklisted() -{ - docker run --rm -it --entrypoint=find $DOCKER_FROM -iname "*.so.*" \ - | xargs -i basename {} | sed 's/\r//g' | sed 's/\(so\.[0-9]\+\).*$/\1/' \ - | sort | uniq > BLACKLISTED_FILES - - for FILE in $(cat BLACKLISTED_FILES | tr "\r\n" " ") ; do - FILES="$(find . -name "${FILE}*" -not -path "./usr/optional/*")" - for FOUND in $FILES ; do - rm -vf "$FOUND" "$(readlink -f "$FOUND")" - done - done - - rm BLACKLISTED_FILES - - # Do not bundle developer stuff - rm -rf usr/include || true - rm -rf usr/lib/cmake || true - rm -rf usr/lib/pkgconfig || true - find . -name '*.la' | xargs -i rm {} -} - diff --git a/buildScripts/getBuildToolsAlpine b/buildScripts/getBuildToolsAlpine new file mode 100755 index 0000000..c58deb3 --- /dev/null +++ b/buildScripts/getBuildToolsAlpine @@ -0,0 +1,30 @@ +#!/bin/sh + +# This shell script automates getting the required build tools (apt install) + +# set the shell environment variable 'UNATTENDED' to '--assume-yes' for +# unattended use (for example in the .travis.yml script) + +echo "" +echo "-------------------------------------------------------------------" +echo "INSTALLING Build Tools (using APK / Alpine)" +echo "-------------------------------------------------------------------" +echo "" + +set -ev + +sudo apk update +sudo apk add \ + sudo \ + wget \ + git \ + pkgconfig \ + ruby \ + cmake \ + make \ + gcc \ + g++ \ + gettext \ + openjdk8 \ + jq + diff --git a/buildScripts/getBuildTools b/buildScripts/getBuildToolsApt similarity index 77% rename from buildScripts/getBuildTools rename to buildScripts/getBuildToolsApt index 41a9b7e..2c6b804 100755 --- a/buildScripts/getBuildTools +++ b/buildScripts/getBuildToolsApt @@ -1,8 +1,8 @@ -#!/bin/bash +#!/bin/sh -# This bash script automates getting the required build tools (apt install) +# This shell script automates getting the required build tools (apt install) -# set the bash environment variable 'UNATTENDED' to '--assume-yes' for +# set the shell environment variable 'UNATTENDED' to '--assume-yes' for # unattended use (for example in the .travis.yml script) echo "" @@ -16,7 +16,8 @@ set -ev sudo apt-get update sudo apt-get $UNATTENDED install \ - aptitude \ + sudo \ + wget \ git \ pkg-config \ ruby \ @@ -30,6 +31,5 @@ sudo apt-get $UNATTENDED install \ dpkg-dev \ gettext \ openjdk-8-jre-headless \ - jq \ - tree + jq diff --git a/buildScripts/getDevLibrariesAlpine b/buildScripts/getDevLibrariesAlpine new file mode 100755 index 0000000..f2a6ec2 --- /dev/null +++ b/buildScripts/getDevLibrariesAlpine @@ -0,0 +1,28 @@ +#!/bin/sh + +# This shell script automates getting the development libraries required to +# build poppler and fontforge + +# set the shell environment variable 'UNATTENDED' to '--assume-yes' for +# unattended use (for example in the .travis.yml script) + +echo "" +echo "-------------------------------------------------------------------" +echo "INSTALLING development libraries (using APK on Alpine)" +echo "-------------------------------------------------------------------" +echo "" + +set -ev + +sudo apk update +sudo apk add \ + gettext \ + gnu-libiconv-dev \ + cairo-dev \ + libpng-dev \ + freetype-dev \ + gettext-dev \ + glib-dev \ + fontconfig-dev \ + libjpeg-turbo-dev \ + libxml2-dev diff --git a/buildScripts/getDevLibraries b/buildScripts/getDevLibrariesApt similarity index 77% rename from buildScripts/getDevLibraries rename to buildScripts/getDevLibrariesApt index b19f8a7..2b6fbce 100755 --- a/buildScripts/getDevLibraries +++ b/buildScripts/getDevLibrariesApt @@ -1,9 +1,9 @@ -#!/bin/bash +#!/bin/sh -# This bash script automates getting the development libraries required to +# This shell script automates getting the development libraries required to # build poppler and fontforge -# set the bash environment variable 'UNATTENDED' to '--assume-yes' for +# set the shell environment variable 'UNATTENDED' to '--assume-yes' for # unattended use (for example in the .travis.yml script) echo "" diff --git a/buildScripts/getFontforge b/buildScripts/getFontforge index c947a20..eebc9e4 100755 --- a/buildScripts/getFontforge +++ b/buildScripts/getFontforge @@ -1,8 +1,9 @@ -#!/bin/bash +#!/bin/sh -# This bash script gets and unpacks the latest fontforge AppImage +# This shell script gets and unpacks the latest fontforge AppImage -source buildScripts/reSourceVersionEnvs +# source buildScripts/reSourceVersionEnvs +. buildScripts/reSourceVersionEnvs echo "" echo "-------------------------------------------------------------------" diff --git a/buildScripts/getPdf2htmlEX b/buildScripts/getPdf2htmlEX deleted file mode 100755 index 3084d1a..0000000 --- a/buildScripts/getPdf2htmlEX +++ /dev/null @@ -1,25 +0,0 @@ -#!/bin/bash - -# This bash script automates the process of getting the original pdf2htmlEX -# source - -PDF2HTMLEX_BRANCH=update-poppler - -echo "" -echo "-------------------------------------------------------------------" -echo "GETTING pdf2htmlEX sources (using wget)" -echo " (PDF2HTMLEX_BRANCH: [$PDF2HTMLEX_BRANCH])" -echo "-------------------------------------------------------------------" -echo "" - -set -ev - -wget https://codeload.github.com/stephengaito/pdf2htmlEX/zip/$PDF2HTMLEX_BRANCH - -mv $PDF2HTMLEX_BRANCH $PDF2HTMLEX_BRANCH.zip - -sudo apt install unzip - -unzip $PDF2HTMLEX_BRANCH.zip - -mv pdf2htmlEX-$PDF2HTMLEX_BRANCH pdf2htmlEX diff --git a/buildScripts/getPoppler b/buildScripts/getPoppler index b5d9403..1179b4d 100755 --- a/buildScripts/getPoppler +++ b/buildScripts/getPoppler @@ -1,8 +1,9 @@ -#!/bin/bash +#!/bin/sh -# This bash script gets and unpacks the latest Poppler source code +# This shell script gets and unpacks the latest Poppler source code -source buildScripts/reSourceVersionEnvs +# source buildScripts/reSourceVersionEnvs +. buildScripts/reSourceVersionEnvs echo "" echo "-------------------------------------------------------------------" diff --git a/buildScripts/installPdf2htmlEX b/buildScripts/installPdf2htmlEX index 5ed22aa..29061ce 100755 --- a/buildScripts/installPdf2htmlEX +++ b/buildScripts/installPdf2htmlEX @@ -1,4 +1,4 @@ -#!/bin/bash +#!/bin/sh echo "" echo "-------------------------------------------------------------------" diff --git a/buildScripts/listFilesByChangeTime b/buildScripts/listFilesByChangeTime index e88213f..91b053c 100755 --- a/buildScripts/listFilesByChangeTime +++ b/buildScripts/listFilesByChangeTime @@ -1,4 +1,4 @@ -#!/bin/bash +#!/bin/sh # see:https://stackoverflow.com/a/7448828 diff --git a/buildScripts/reportEnvs b/buildScripts/reportEnvs index d49fedf..b24a198 100755 --- a/buildScripts/reportEnvs +++ b/buildScripts/reportEnvs @@ -1,8 +1,12 @@ -#!/bin/bash +#!/bin/sh -# This bash script reports the more important TRAVIS environment variables +# This shell script reports the more important TRAVIS environment variables + +# source ./buildScripts/reSourceVersionEnvs +. ./buildScripts/reSourceVersionEnvs echo "TravisCI env:" +echo " dist: [$TRAVIS_DIST]" echo " tag: [$TRAVIS_TAG]" echo " branch: [$TRAVIS_BRANCH]" echo " commit: [$TRAVIS_COMMIT]" @@ -13,17 +17,20 @@ echo "Build env:" echo " prefix: [$PDF2HTMLEX_PREFIX]" echo " unattended: [$UNATTENDED]" echo " make parallel: [$MAKE_PARALLEL]" -source ./buildScripts/reSourceVersionEnvs +echo " distribution: [$BUILD_DIST]" echo "pdf2htmlEX env:" +echo " version: [$PDF2HTMLEX_VERSION]" echo " poppler: [$POPPLER_VERSION]" echo " fontforge: [$FONTFORGE_VERSION]" echo " pdf2htmlEX: [$PDF2HTMLEX_BRANCH]" echo " machine arch: [$MACHINE_ARCH]" echo "release env:" +echo " build date: [$BUILD_DATE]" echo " build time: [$BUILD_TIME]" echo " docker from: [$DOCKER_FROM]" echo " docker name: [$DOCKER_NAME]" echo " appImage name: [$APPIMAGE_NAME]" +echo " dpkg name: [$DPKG_NAME]" echo "uploadTool suffix: [$UPLOADTOOL_SUFFIX]" diff --git a/buildScripts/runTests b/buildScripts/runTests new file mode 100755 index 0000000..b7d4f8b --- /dev/null +++ b/buildScripts/runTests @@ -0,0 +1,33 @@ +#!/bin/sh + +# source ./buildScripts/reSourceVersionEnvs +. ./buildScripts/reSourceVersionEnvs + +echo "" +echo "-------------------------------------------------------------------" +echo "RUNNING pdf2htmlEX tests" +echo "-------------------------------------------------------------------" +echo "" + +export TEST_RESULTS_NAME="pdf2htmlEX-$PDF2HTMLEX_BRANCH-$BUILD_TIME-$BUILD_DIST-$MACHINE_ARCH-testResults.zip" +echo "export TEST_RESULTS_NAME=\"$TEST_RESULTS_NAME\"" >> buildScripts/reSourceVersionEnvs + +set -ev + +# This shell script runs the pdf2htmlEX tests + +cd pdf2htmlEX/test + +# The following is only needed for the local browser tests +# +./installAutomaticTestSoftwareApt + +./runLocalTestsShell + +./runLocalBrowserTests + +cd ../.. + +mkdir -p imageBuild + +zip -r imageBuild/$TEST_RESULTS_NAME /tmp/pdf2htmlEX/html /tmp/pdf2htmlEX/png diff --git a/buildScripts/travisLinuxDoItAll b/buildScripts/travisLinuxDoItAll index 13577c9..4d3e697 100755 --- a/buildScripts/travisLinuxDoItAll +++ b/buildScripts/travisLinuxDoItAll @@ -1,16 +1,19 @@ -#!/bin/bash +#!/bin/sh -# This bash script builds everyting on an TravisCI Linux (Ubunutu) worker +# This shell script builds everyting on an TravisCI Linux (Ubunutu) worker set -ev export UNATTENDED="--assume-yes" + export MAKE_PARALLEL="-j $(nproc)" -export PDF2HTMLEX_BRANCH="$(git rev-parse --abbrev-ref HEAD)" +export PDF2HTMLEX_BRANCH=$TRAVIS_BRANCH export PDF2HTMLEX_PREFIX=/usr/local +export PDF2HTMLEX_PATH=/usr/local/bin/pdf2htmlEX + # Ensure all Apt packages are installed with no user interaction # export DEBIAN_FRONTEND=noninteractive @@ -20,17 +23,17 @@ export DEBIAN_FRONTEND=noninteractive ./buildScripts/versionEnvs ./buildScripts/reportEnvs -./buildScripts/getBuildTools -./buildScripts/getDevLibraries +./buildScripts/getBuildToolsApt +./buildScripts/getDevLibrariesApt ./buildScripts/getPoppler ./buildScripts/buildPoppler ./buildScripts/getFontforge ./buildScripts/buildFontforge ./buildScripts/buildPdf2htmlEX ./buildScripts/installPdf2htmlEX -#./testScripts/runLocalTests +./buildScripts/runTests ./buildScripts/createAppImage ./buildScripts/createDebianPackage -./buildScripts/createDockerImage +./buildScripts/createDockerUbuntuImageFromDeb #./buildScripts/uploadGitHubRelease #./buildScripts/uploadDockerImage diff --git a/buildScripts/uploadDockerImage b/buildScripts/uploadDockerImage index 13fe66f..48cb0ec 100755 --- a/buildScripts/uploadDockerImage +++ b/buildScripts/uploadDockerImage @@ -1,6 +1,6 @@ -#!/bin/bash +#!/bin/sh -# This bash script uploads the pdf2htmlEX docker image to docker hub +# This shell script uploads the pdf2htmlEX docker image to docker hub # # We EXPECT the following environment variables to be set: # DOCKER_USERNAME @@ -9,7 +9,8 @@ # DOCKER_PASSWORD (if not set you will be asked for your password) # -source ./buildScripts/reSourceVersionEnvs +# source ./buildScripts/reSourceVersionEnvs +. ./buildScripts/reSourceVersionEnvs echo "" echo "-------------------------------------------------------------------" diff --git a/buildScripts/uploadGitHubRelease b/buildScripts/uploadGitHubRelease index b0d23bb..ffde585 100755 --- a/buildScripts/uploadGitHubRelease +++ b/buildScripts/uploadGitHubRelease @@ -1,6 +1,6 @@ -#!/bin/bash +#!/bin/sh -# This bash script uploads the pdf2htmlEX release artefacts +# This shell script uploads the pdf2htmlEX release artefacts # # We EXPECT the following environment variables to be set: # GITHUB_TOKEN @@ -10,9 +10,11 @@ # DOCKER_PASSWORD (if not set you will be asked for your password) # -source ./buildScripts/reSourceVersionEnvs -cp ./buildScripts/reSourceVersionEnvs imageBuild/buildInfo.sh -source ./buildScripts/uploadGitHubReleaseDSL +# source ./buildScripts/reSourceVersionEnvs +. ./buildScripts/reSourceVersionEnvs +cp ./buildScripts/reSourceVersionEnvs imageBuild/buildInfo.sh +. ./buildScripts/uploadGitHubReleaseDSL +# source ./buildScripts/uploadGitHubReleaseDSL echo "" echo "-------------------------------------------------------------------" @@ -60,6 +62,8 @@ fi cd imageBuild +ls -la + echo "machine api.github.com" > $HOME/.netrc echo " login $GITHUB_USERNAME" >> $HOME/.netrc echo " password $GITHUB_TOKEN" >> $HOME/.netrc @@ -67,16 +71,22 @@ echo "machine uploads.github.com" >> $HOME/.netrc echo " login $GITHUB_USERNAME" >> $HOME/.netrc echo " password $GITHUB_TOKEN" >> $HOME/.netrc -echo $APPIMAGE_NAME > appImageName.txt -echo $DOCKER_NAME > dockerImageName.txt +echo $APPIMAGE_NAME > appImageName.txt +echo $DOCKER_NAME > dockerImageName.txt +echo $DPKG_NAME > debianArchiveName.txt +echo $TEST_RESULTS_NAME > testResultsName.txt -deleteReleaseByTag "$TRAVIS_REPO_SLUG" "continuous" +deleteReleaseByTag "$TRAVIS_REPO_SLUG" "continuous-$BUILD_DIST" -createNewRelease "$TRAVIS_REPO_SLUG" "continuous" "Latest release" \ +createNewRelease "$TRAVIS_REPO_SLUG" "continuous-$BUILD_DIST" "Latest $BUILD_DIST release" \ ../buildScripts/uploadGitHubReleaseMessage -uploadAnAsset $upload_url "appImageName.txt" "text/plain" -uploadAnAsset $upload_url "buildInfo.sh" "text/plain" -uploadAnAsset $upload_url "dockerImageName.txt" "text/plain" -uploadAnAsset $upload_url $APPIMAGE_NAME "application/zip" +uploadAnAsset $upload_url "appImageName.txt" "text/plain" +uploadAnAsset $upload_url "testResultsName.txt" "text/plain" +uploadAnAsset $upload_url "buildInfo.sh" "text/plain" +uploadAnAsset $upload_url "dockerImageName.txt" "text/plain" +uploadAnAsset $upload_url "debianArchiveName.txt" "text/plain" +uploadAnAsset $upload_url $APPIMAGE_NAME "application/zip" +uploadAnAsset $upload_url $TEST_RESULTS_NAME "application/zip" +uploadAnAsset $upload_url $DPKG_NAME "application/x-debian-package" diff --git a/buildScripts/uploadGitHubReleaseDSL b/buildScripts/uploadGitHubReleaseDSL index 3a0207a..eb226d6 100755 --- a/buildScripts/uploadGitHubReleaseDSL +++ b/buildScripts/uploadGitHubReleaseDSL @@ -1,8 +1,8 @@ -#!/usr/bin/env bash +#!/usr/bin/env sh -# This bash script is a collection of functions to interact with the GitHub -# release and git/tag APIs. It essentially provides a bash based DSL for -# uploading GitHub release artefacts. +# This shell script is a collection of functions to interact with the GitHub +# release and git/tag APIs. It essentially provides a shell script based DSL +# for uploading GitHub release artefacts. # NOTE-1: These functions make use of your ".netrc" file to authenticate # with GitHub. In order to use these functions, make sure you have **both** @@ -171,7 +171,7 @@ function uploadAnAsset { echo "" } -# The above has been heavily modified into a collection of bash functions +# The above has been heavily modified into a collection of shell functions # by Stephen Gaito working on the pdf2htmlEX project. # # It has been based upon the code in: diff --git a/buildScripts/uploadGitHubReleaseMessage b/buildScripts/uploadGitHubReleaseMessage index 9c83d0d..0070e94 100644 --- a/buildScripts/uploadGitHubReleaseMessage +++ b/buildScripts/uploadGitHubReleaseMessage @@ -1,6 +1,35 @@ -This is the development/testing release of the pdf2htmlEX AppImage. +This is the most recent release of the pdf2htmlEX project. -You can download the AppImage and 'just run it'. +We release three types of binary objects: -See: https://appimage.org/ for details +1. The file *.AppImage on this release page is an AppImage + You can download the AppImage, make it executable and then 'just run + it'. + + See: https://appimage.org/ for details + +2. The file *.deb is a Debian Archive + + You can download the Debian Archive and install it locally using: + + sudo apt install <> + + While the Debian archive is built on a specific Ubuntu release, it can + usually be used on any more recent Ubuntu releases. + +3. You can alternatively get pdf2htmlEX Docker images from Docker Hub in + either of the repositories: + + pdf2htmlEX/pdf2htmlEX (offical release) + + or + + stephengaito/pdf2htmlEX (development releases) + +We also release a zip archive of the browser test results in the +*-testResults.zip file. + +Finally, the various *.txt and *buildInfo.sh files have well known +(unchanging) names which can be used by automatic scripts to identifiy the +current full names of the corresponding binary artifacts. diff --git a/buildScripts/uploadImages b/buildScripts/uploadImages index 7adbe9c..28760a0 100755 --- a/buildScripts/uploadImages +++ b/buildScripts/uploadImages @@ -1,6 +1,6 @@ -#!/bin/bash +#!/bin/sh -# This bash script uploads already existing pdf2htmlEX AppImage and Docker +# This shell script uploads already existing pdf2htmlEX AppImage and Docker # Images ################ diff --git a/buildScripts/versionEnvs b/buildScripts/versionEnvs index 11a4f69..00ac4b6 100755 --- a/buildScripts/versionEnvs +++ b/buildScripts/versionEnvs @@ -1,11 +1,13 @@ -#!/bin/bash +#!/bin/sh -# This bash script exports environment variables for the latest software +# This shell script exports environment variables for the latest software # versions # see: https://poppler.freedesktop.org/releases.html # current working: 0.83.0 +export PDF2HTMLEX_VERSION=0.18.8.alpha + export POPPLER_VERSION=poppler-0.89.0 #export POPPLER_VERSION=poppler-0.88.0 #export POPPLER_VERSION=poppler-0.87.0 @@ -40,13 +42,37 @@ if [ -z "$PDF2HTMLEX_BRANCH" ]; then fi fi -export MACHINE_ARCH="$(uname -m)" +if test -r /etc/lsb-release ; then + cat /etc/lsb-release + # source /etc/lsb-release + . /etc/lsb-release + export BUILD_DIST=$DISTRIB_CODENAME + export BUILD_OS=$DISTRIB_ID +elif test -r /etc/alpine-release ; then + cat /etc/alpine-release + export BUILD_OS=alpine + export BUILD_DIST=$(cat /etc/alpine-release) +else + echo "FAILURE: could not determine release" + exit -1 +fi + +export BUILD_DATE="$(date +%Y%m%d)" export BUILD_TIME="$(date +%Y_%m_%d-%H_%M_%S)" -echo "export POPPLER_VERSION=\"$POPPLER_VERSION\"" > buildScripts/reSourceVersionEnvs -echo "export FONTFORGE_VERSION=\"$FONTFORGE_VERSION\"" >> buildScripts/reSourceVersionEnvs -echo "export PDF2HTMLEX_BRANCH=\"$PDF2HTMLEX_BRANCH\"" >> buildScripts/reSourceVersionEnvs -echo "export MACHINE_ARCH=\"$MACHINE_ARCH\"" >> buildScripts/reSourceVersionEnvs -echo "export BUILD_TIME=\"$BUILD_TIME\"" >> buildScripts/reSourceVersionEnvs -echo "export PDF2HTMLEX_PREFIX=\"$PDF2HTMLEX_PREFIX\"" >> buildScripts/reSourceVersionEnvs +export MACHINE_ARCH="$(uname -m)" + +export PDF2HTMLEX_NAME=$PDF2HTMLEX_VERSION-$PDF2HTMLEX_BRANCH-$BUILD_DATE-$BUILD_OS-$BUILD_DIST-$MACHINE_ARCH + +echo "export PDF2HTMLEX_NAME=\"$PDF2HTMLEX_NAME\"" > buildScripts/reSourceVersionEnvs +echo "export PDF2HTMLEX_VERSION=\"$PDF2HTMLEX_VERSION\"" >> buildScripts/reSourceVersionEnvs +echo "export POPPLER_VERSION=\"$POPPLER_VERSION\"" >> buildScripts/reSourceVersionEnvs +echo "export FONTFORGE_VERSION=\"$FONTFORGE_VERSION\"" >> buildScripts/reSourceVersionEnvs +echo "export PDF2HTMLEX_BRANCH=\"$PDF2HTMLEX_BRANCH\"" >> buildScripts/reSourceVersionEnvs +echo "export MACHINE_ARCH=\"$MACHINE_ARCH\"" >> buildScripts/reSourceVersionEnvs +echo "export BUILD_OS=\"$BUILD_OS\"" >> buildScripts/reSourceVersionEnvs +echo "export BUILD_DIST=\"$BUILD_DIST\"" >> buildScripts/reSourceVersionEnvs +echo "export BUILD_DATE=\"$BUILD_DATE\"" >> buildScripts/reSourceVersionEnvs +echo "export BUILD_TIME=\"$BUILD_TIME\"" >> buildScripts/reSourceVersionEnvs +echo "export PDF2HTMLEX_PREFIX=\"$PDF2HTMLEX_PREFIX\"" >> buildScripts/reSourceVersionEnvs diff --git a/pdf2htmlEX/CMakeLists.txt b/pdf2htmlEX/CMakeLists.txt index 691ead1..218c54d 100644 --- a/pdf2htmlEX/CMakeLists.txt +++ b/pdf2htmlEX/CMakeLists.txt @@ -79,9 +79,18 @@ set(FONTFORGE_LIBRARIES ${FONTFORGE_LIBRARIES} ${CMAKE_SOURCE_DIR}/../fontforge/build/lib/libfontforge.a ) +# If we are using Alpine Linux then we need to add -lintl +# +if (EXISTS /usr/lib/libintl.so ) + set(LIB_INTL_LIBRARIES -lintl ) +else () + set(LIB_INTL_LIBRARIES "" ) +endif() + set(PDF2HTMLEX_LIBS ${PDF2HTMLEX_LIBS} ${POPPLER_LIBRARIES} ${FONTFORGE_LIBRARIES} + ${LIB_INTL_LIBRARIES} ${CAIRO_LIBRARIES} -ljpeg -lpng @@ -262,6 +271,7 @@ set(PDF2HTMLEX_TMPDIR /tmp/pdf2htmlEX/tmp) set(PDF2HTMLEX_DATDIR /tmp/pdf2htmlEX/dat) set(PDF2HTMLEX_PNGDIR /tmp/pdf2htmlEX/png) set(PDF2HTMLEX_OUTDIR /tmp/pdf2htmlEX/out) +set(PDF2HTMLEX_HTMDIR /tmp/pdf2htmlEX/html) file(MAKE_DIRECTORY ${PDF2HTMLEX_TMPDIR}) file(MAKE_DIRECTORY ${PDF2HTMLEX_DATDIR}) file(MAKE_DIRECTORY ${PDF2HTMLEX_PNGDIR}) diff --git a/pdf2htmlEX/pdf2htmlEX.1.in b/pdf2htmlEX/pdf2htmlEX.1.in index d3cd881..570997c 100644 --- a/pdf2htmlEX/pdf2htmlEX.1.in +++ b/pdf2htmlEX/pdf2htmlEX.1.in @@ -359,7 +359,7 @@ pdf2htmlEX is written by Lu Wang .SH SEE ALSO .TP Home page -https://github.com/coolwanglu/pdf2htmlEX +https://github.com/pdf2htmlEX/pdf2htmlEX .TP pdf2htmlEX Wiki -https://github.com/coolwanglu/pdf2htmlEX/wiki +https://github.com/pdf2htmlEX/pdf2htmlEX/wiki diff --git a/pdf2htmlEX/share/base.css.in b/pdf2htmlEX/share/base.css.in index 515134d..7b2feac 100644 --- a/pdf2htmlEX/share/base.css.in +++ b/pdf2htmlEX/share/base.css.in @@ -2,7 +2,7 @@ /*! * Base CSS for pdf2htmlEX * Copyright 2012,2013 Lu Wang - * https://github.com/coolwanglu/pdf2htmlEX/blob/master/share/LICENSE + * https://github.com/pdf2htmlEX/pdf2htmlEX/blob/master/share/LICENSE */ /* Part 1: Web Page Layout: Free to modify, except for a few of them which are required by pdf2htmlEX.js, see the comments */ #sidebar { /* Sidebar */ diff --git a/pdf2htmlEX/share/fancy.css.in b/pdf2htmlEX/share/fancy.css.in index bdba3f6..138b6f2 100644 --- a/pdf2htmlEX/share/fancy.css.in +++ b/pdf2htmlEX/share/fancy.css.in @@ -2,7 +2,7 @@ /*! * Fancy styles for pdf2htmlEX * Copyright 2012,2013 Lu Wang - * https://github.com/coolwanglu/pdf2htmlEX/blob/master/share/LICENSE + * https://github.com/pdf2htmlEX/pdf2htmlEX/blob/master/share/LICENSE */ @keyframes fadein { from { opacity:0;} to { opacity:1;} } @-webkit-keyframes fadein { from { opacity:0;} to { opacity:1;} } diff --git a/pdf2htmlEX/share/manifest b/pdf2htmlEX/share/manifest index b03560f..0380867 100644 --- a/pdf2htmlEX/share/manifest +++ b/pdf2htmlEX/share/manifest @@ -17,7 +17,7 @@ # Declaration - Do not modify """ - + diff --git a/pdf2htmlEX/share/pdf2htmlEX.js.in b/pdf2htmlEX/share/pdf2htmlEX.js.in index a678759..9423ce9 100644 --- a/pdf2htmlEX/share/pdf2htmlEX.js.in +++ b/pdf2htmlEX/share/pdf2htmlEX.js.in @@ -2,7 +2,7 @@ /** * @license pdf2htmlEX.js: Core UI functions for pdf2htmlEX * Copyright 2012,2013 Lu Wang and other contributors - * https://github.com/coolwanglu/pdf2htmlEX/blob/master/share/LICENSE + * https://github.com/pdf2htmlEX/pdf2htmlEX/blob/master/share/LICENSE */ /* diff --git a/pdf2htmlEX/test/README.md b/pdf2htmlEX/test/README.md index 7b8f708..24a0209 100644 --- a/pdf2htmlEX/test/README.md +++ b/pdf2htmlEX/test/README.md @@ -19,7 +19,7 @@ and needs to be fixed. Rerun `runLocalBrowserTests` and use the ## Running tests -There are three bash scripts which automate the running of a given +There are three shell scripts which automate the running of a given collection of tests: 1. **runLocalTests** runs a simple collection of tests which do not diff --git a/pdf2htmlEX/test/browser_tests.py b/pdf2htmlEX/test/browser_tests.py index 74634fa..e09c54a 100644 --- a/pdf2htmlEX/test/browser_tests.py +++ b/pdf2htmlEX/test/browser_tests.py @@ -37,13 +37,23 @@ class BrowserTests(Common): ref_htmlfolder = os.path.join(self.TEST_DATA_DIR, basefilename) ref_htmlfilename = os.path.join(ref_htmlfolder, htmlfilename) out_htmlfilename = os.path.join(self.OUTDIR, htmlfilename) - - pdf2htmlEX_args = self.DEFAULT_PDF2HTMLEX_ARGS + args + [ + pre_htmlfilename = os.path.join(self.PREDIR, htmlfilename) + + try: + # see if we have pre-compiled the html file... + # if so simply copy it into place + # + shutil.copy(pre_htmlfilename, out_htmlfilename) + except: + # we have not pre-compiled the html file + # so create it using pdf2htmlEX + # + pdf2htmlEX_args = self.DEFAULT_PDF2HTMLEX_ARGS + args + [ os.path.join(self.TEST_DATA_DIR, filename), htmlfilename ] - result = self.run_pdf2htmlEX(pdf2htmlEX_args) - - self.assertIn(htmlfilename, result['output_files'], 'HTML file is not generated') + result = self.run_pdf2htmlEX(pdf2htmlEX_args) + # + self.assertIn(htmlfilename, result['output_files'], 'HTML file is not generated') if self.GENERATING_MODE: # copy generated html files @@ -51,6 +61,20 @@ class BrowserTests(Common): shutil.copytree(self.OUTDIR, ref_htmlfolder) return + # keep a record of the HTML files (and any differences) + # for later reporting of test results + # + try: + os.makedirs(self.HTMDIR, 0o755, True) + outHtmlFile = os.path.join(self.HTMDIR, basefilename+'.out.html') + refHtmlFile = os.path.join(self.HTMDIR, basefilename+'.ref.html') + difHtmlFile = os.path.join(self.HTMDIR, basefilename+'.diff.html') + shutil.copy(out_htmlfilename, outHtmlFile) + shutil.copy(ref_htmlfilename, refHtmlFile) + os.system("diff "+outHtmlFile+" "+refHtmlFile+" > "+difHtmlFile+" 2>&1") + except: + pass + pngfilename_out = os.path.join(self.PNGDIR, basefilename + '.out.png') self.generate_image(out_htmlfilename, pngfilename_out) out_img = Image.open(pngfilename_out).convert('RGB') @@ -70,9 +94,11 @@ class BrowserTests(Common): print("\nTesting at: [", basefilename, "]") if diff_bbox is None: - print(" passed") + print(" SUCCESS: ", basefilename) + elif basefilename == "test_fail" : + print(" SUCCESS: ", basefilename) else: - print(" diff bounding box: ", diff_bbox, " should be None") + print(" FAILURE: ", basefilename, " diff bounding box: ", diff_bbox, " should be None") diff_size = (diff_bbox[2] - diff_bbox[0]) * (diff_bbox[3] - diff_bbox[1]) img_size = ref_img.size[0] * ref_img.size[1] self.fail(('PNG files %s and %s differ by at most %d pixels, '+ diff --git a/pdf2htmlEX/test/installAutomaticTestSoftware b/pdf2htmlEX/test/installAutomaticTestSoftwareApt similarity index 76% rename from pdf2htmlEX/test/installAutomaticTestSoftware rename to pdf2htmlEX/test/installAutomaticTestSoftwareApt index 891dcd5..fd41a7f 100755 --- a/pdf2htmlEX/test/installAutomaticTestSoftware +++ b/pdf2htmlEX/test/installAutomaticTestSoftwareApt @@ -1,6 +1,8 @@ -#!/bin/bash +#!/bin/sh -# This bash script installs all local software required to run the +set -ev + +# This shell script installs all local software required to run the # pdf2htmlEX tests export DEBIAN_FRONTEND=noninteractive @@ -8,6 +10,9 @@ export DEBIAN_FRONTEND=noninteractive # Start by making sure all required apt packages exist # sudo apt -y install \ + wget \ + diffutils \ + zip \ python3 \ python3-pip \ xvfb \ @@ -15,7 +20,8 @@ sudo apt -y install \ # Now get the geckodriver for firefox (as required by selenium) # -pushd /tmp +oldPWD=$(pwd) +cd /tmp # wget https://github.com/mozilla/geckodriver/releases/download/v0.26.0/geckodriver-v0.26.0-linux64.tar.gz # @@ -23,7 +29,7 @@ tar xvf geckodriver-v0.26.0-linux64.tar.gz # sudo mv geckodriver /usr/local/bin # -popd +cd $oldPWD # Now make sure all python packages exist (install into the local user's # PyPI archive) diff --git a/pdf2htmlEX/test/installManualTestSoftware b/pdf2htmlEX/test/installManualTestSoftware index d7c8ab3..5dc3e8a 100755 --- a/pdf2htmlEX/test/installManualTestSoftware +++ b/pdf2htmlEX/test/installManualTestSoftware @@ -1,6 +1,6 @@ -#!/bin/bash +#!/bin/sh -# This bash script installs all local software required to run the +# This shell script installs all local software required to run the # pdf2htmlEX tests export DEBIAN_FRONTEND=noninteractive diff --git a/pdf2htmlEX/test/produceHtmlForBrowserTests b/pdf2htmlEX/test/produceHtmlForBrowserTests new file mode 100755 index 0000000..448a35f --- /dev/null +++ b/pdf2htmlEX/test/produceHtmlForBrowserTests @@ -0,0 +1,111 @@ +#!/bin/sh + +# This shell script walks through the browser_tests directory running +# pdf2htmlEX on each *.pdf file. + +# This is how we run pdf2htmlEX on a particular file, and arguments. +# +runPdf2htmlEX() { + pdfFileName=$1 + htmlFileName=$(echo $pdfFileName | cut -d'.' -f1).html + arguments=$2 + # + echo "" + echo "---" + echo " pdfFileName: [$pdfFileName]" + echo "htmlFileName: [$htmlFileName]" + echo " arguments: [$arguments]" + # + # now run pdf2htmlEX to produce the output files + # + echo $PDF2HTMLEX_PATH \ + --data-dir=$PDF2HTMLEX_DATDIR \ + --dest-dir $PDF2HTMLEX_TMPDIR \ + --fit-width=800 --last-page=1 \ + $arguments \ + browser_tests/$pdfFileName \ + $htmlFileName + # + $PDF2HTMLEX_PATH \ + --data-dir=$PDF2HTMLEX_DATDIR \ + --dest-dir $PDF2HTMLEX_TMPDIR \ + --fit-width=800 --last-page=1 \ + $arguments \ + browser_tests/$pdfFileName \ + $htmlFileName +} + +# This is how we copy a file omitting lines between '#TEST_IGNORE_BEGIN' +# and '#TEST_IGNORE_END' (we pipe the file in via stdin and save it via +# stdout) +# +copy_TEST_IGNORE_file() { + skipLine=echo + while IFS= read -r line ; do + if echo $line | grep -q "^#TEST_IGNORE_BEGIN" ; then + skipLine=true + elif echo $line | grep -q "^#TEST_IGNORE_END" ; then + skipLine=echo + else + $skipLine "$line" + fi + done +} + +if test -z "$PDF2HTMLEX_PATH" ; then + echo "PANIC: we do not know where to find the pdf2htmlEX executable" + exit 1 +fi + +if test -z "$PDF2HTMLEX_DATDIR" ; then + export PDF2HTMLEX_DATDIR=/tmp/pdf2htmlEX/dat +fi + +if test -z "$PDF2HTMLEX_TMPDIR" ; then + export PDF2HTMLEX_TMPDIR=/tmp/pdf2htmlEX/tmp +fi + +if test -z "$PDF2HTMLEX_PREDIR" ; then + export PDF2HTMLEX_PREDIR=/tmp/pdf2htmlEX/pre +fi + +if test -z "$PDF2HTMLEX_TEST_DIR" ; then + export PDF2HTMLEX_TEST_DIR=. +fi + +# clear out the TMPDIR +# +rm -rf $PDF2HTMLEX_TMPDIR +mkdir -p $PDF2HTMLEX_TMPDIR +# + +# setup the correct data files +# +cat $PDF2HTMLEX_TEST_DIR/../share/manifest | \ + copy_TEST_IGNORE_file > $PDF2HTMLEX_DATDIR/manifest + +cp $PDF2HTMLEX_TEST_DIR/../share/base.min.css $PDF2HTMLEX_DATDIR +cp $PDF2HTMLEX_TEST_DIR/../test/fancy.min.css $PDF2HTMLEX_DATDIR + +runPdf2htmlEX 'test_fail.pdf' + +runPdf2htmlEX 'basic_text.pdf' + +runPdf2htmlEX 'geneve_1564.pdf' + +runPdf2htmlEX 'text_visibility.pdf' '--correct-text-visibility=1' + +runPdf2htmlEX 'with_form.pdf' '--process-form=1' + +runPdf2htmlEX 'invalid_unicode_issue477.pdf' + +runPdf2htmlEX 'svg_background_with_page_rotation_issue402.pdf' '--bg-format=svg' + +runPdf2htmlEX 'fontfile3_opentype.pdf' + +# clear out the PREDIR +# +rm -rf $PDF2HTMLEX_PREDIR +mkdir -p $PDF2HTMLEX_PREDIR +# +cp $PDF2HTMLEX_TMPDIR/* $PDF2HTMLEX_PREDIR diff --git a/pdf2htmlEX/test/runLocalBrowserTests b/pdf2htmlEX/test/runLocalBrowserTests index 4d122c7..f27cae2 100755 --- a/pdf2htmlEX/test/runLocalBrowserTests +++ b/pdf2htmlEX/test/runLocalBrowserTests @@ -1,8 +1,6 @@ -#!/bin/bash +#!/bin/sh -set -ev - -# This bash script runs the local browser tests +# This shell script runs the local browser tests # We start by running a virtual frame buffer as display 99.0 # @@ -50,4 +48,8 @@ ps -ef | grep -v grep | grep Xvfb echo "-----------------------------------------------------------" echo "" -exit $returnCode +if [ -z "$TRAVIS_DIST"] ; then + exit $returnCode +fi + +exit 0 diff --git a/pdf2htmlEX/test/runLocalTests b/pdf2htmlEX/test/runLocalTests deleted file mode 100755 index f8db9c7..0000000 --- a/pdf2htmlEX/test/runLocalTests +++ /dev/null @@ -1,7 +0,0 @@ -#!/bin/bash - -set -ev - -# This bash script runs the (simple non-browser) tests - -python3 test_output.py diff --git a/pdf2htmlEX/test/runLocalTestsPython b/pdf2htmlEX/test/runLocalTestsPython new file mode 100755 index 0000000..8f0fa85 --- /dev/null +++ b/pdf2htmlEX/test/runLocalTestsPython @@ -0,0 +1,7 @@ +#!/bin/sh + +set -ev + +# This shell script runs the (simple non-browser) tests + +python3 test_output.py diff --git a/pdf2htmlEX/test/runLocalTestsShell b/pdf2htmlEX/test/runLocalTestsShell new file mode 100755 index 0000000..692245a --- /dev/null +++ b/pdf2htmlEX/test/runLocalTestsShell @@ -0,0 +1,7 @@ +#!/bin/sh + +set -ev + +# This shell script runs the (simple non-browser) tests + +./testOutput diff --git a/pdf2htmlEX/test/test.py.in b/pdf2htmlEX/test/test.py.in index 495c4c0..163626f 100755 --- a/pdf2htmlEX/test/test.py.in +++ b/pdf2htmlEX/test/test.py.in @@ -12,9 +12,11 @@ class Common(object): Variables and methods for common use in different tests. """ PDF2HTMLEX_PATH = "@PDF2HTMLEX_PATH@" # defined in CMakeLists.txt - if not os.path.isfile(PDF2HTMLEX_PATH) or not os.access(PDF2HTMLEX_PATH, os.X_OK): - print >> sys.stderr, "Cannot locate pdf2htmlEX executable, expected at ", PDF2HTMLEX_PATH, - ". Make sure source was built before running this test." + PDF2HTMLEX_EXE = PDF2HTMLEX_PATH.split() + PDF2HTMLEX_EXE = PDF2HTMLEX_EXE[0] + if not os.path.isfile(PDF2HTMLEX_EXE) or not os.access(PDF2HTMLEX_EXE, os.X_OK): + print( sys.stderr, "Cannot locate pdf2htmlEX executable, expected at ", PDF2HTMLEX_EXE, + ". Make sure source was built before running this test." ) exit(1) SRC_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) @@ -28,6 +30,8 @@ class Common(object): PNGDIR = "@PDF2HTMLEX_PNGDIR@" DATDIR = "@PDF2HTMLEX_DATDIR@" OUTDIR = "@PDF2HTMLEX_OUTDIR@" + PREDIR = "@PDF2HTMLEX_PREDIR@" + HTMDIR = "@PDF2HTMLEX_HTMDIR@" def setUp(self): # filter manifest @@ -61,11 +65,14 @@ class Common(object): shutil.rmtree(self.TMPDIR, ignore_errors=False, onerror=None) os.mkdir(self.TMPDIR) - args = [Common.PDF2HTMLEX_PATH, - '--data-dir', self.DATDIR, + # To allow the user to specify additional command line arguments + # in the path, we split the PATH on whitespace and then append our + # arguments. + # + args = Common.PDF2HTMLEX_PATH.split() + [ '--data-dir', self.DATDIR, '--dest-dir', self.TMPDIR - ] + args - + ] + args + with open(os.devnull, 'w') as fnull: return_code = subprocess.call(list(map(str, args)), stderr=fnull) diff --git a/pdf2htmlEX/test/testOutput b/pdf2htmlEX/test/testOutput new file mode 100755 index 0000000..3a44f25 --- /dev/null +++ b/pdf2htmlEX/test/testOutput @@ -0,0 +1,206 @@ +#!/bin/sh + +# This shell script checks that pdf2htmlEX does not crash, and produces +# correct files. Do not check the content of the files. + +# We use a shell script to implement the python3 test_output.py so that we +# do not need to install any extra packages to test this functionality. + +# This is how we run pdf2htmlEX on a particular file, and arguments. +# +runPdf2htmlEX() { + # collect the arguments + # + export LAST_TEST_NAME="$*" + # + pdfFileName=$1 + # + shift + arguments=$* + + # clear out the TMPDIR + # + rm -rf $PDF2HTMLEX_TMPDIR + mkdir -p $PDF2HTMLEX_TMPDIR + # + # now run pdf2htmlEX to produce the output files + # + echo "test: [$LAST_TEST_NAME]" + echo "---" + $PDF2HTMLEX_PATH \ + --data-dir $PDF2HTMLEX_DATDIR \ + --dest-dir $PDF2HTMLEX_TMPDIR \ + $PDF2HTMLEX_TEST_DIR/test_output/$pdfFileName \ + $arguments +} + +# This is how we test for expected output files +# +hasExpectedFiles() { + filesFound="true" + for anExpectedFile in $1 ; do + if ! test -r $PDF2HTMLEX_TMPDIR/$anExpectedFile ; then + echo "NOT FOUND [$PDF2HTMLEX_TMPDIR/$anExpectedFile]" + filesFound="false" + fi + done + if test $filesFound = "true" ; then + echo "SUCCESS: $LAST_TEST_NAME" >> testOutputResults + else + echo "FAILURE: $LAST_TEST_NAME" >> testOutputResults + fi + echo "---" + echo "" +} + +# This is how we copy a file omitting lines between '#TEST_IGNORE_BEGIN' +# and '#TEST_IGNORE_END' (we pipe the file in via stdin and save it via +# stdout) +# +copy_TEST_IGNORE_file() { + skipLine=echo + while IFS= read -r line ; do + if echo $line | grep -q "^#TEST_IGNORE_BEGIN" ; then + skipLine=true + elif echo $line | grep -q "^#TEST_IGNORE_END" ; then + skipLine=echo + else + $skipLine "$line" + fi + done +} + +if test -z "$PDF2HTMLEX_PATH" ; then + echo "PANIC: we do not know where to find the pdf2htmlEX executable" + exit 1 +fi + +if test -z "$PDF2HTMLEX_DATDIR" ; then + export PDF2HTMLEX_DATDIR=/tmp/pdf2htmlex/dat +fi + +mkdir -p $PDF2HTMLEX_DATDIR + +if test -z "$PDF2HTMLEX_TMPDIR" ; then + export PDF2HTMLEX_TMPDIR=/tmp/pdf2htmlex/tmp +fi + +mkdir -p $PDF2HTMLEX_TMPDIR + +if test -z "$PDF2HTMLEX_TEST_DIR" ; then + export PDF2HTMLEX_TEST_DIR=. +fi + +mkdir -p $PDF2HTMLEX_TEST_DIR + +# Make sure any previous testOutputResults are cleared +# +rm -f testOutputResults + +# setup the correct data files +# +cat $PDF2HTMLEX_TEST_DIR/../share/manifest | \ + copy_TEST_IGNORE_file > $PDF2HTMLEX_DATDIR/manifest + +cp $PDF2HTMLEX_TEST_DIR/../share/base.min.css $PDF2HTMLEX_DATDIR +cp $PDF2HTMLEX_TEST_DIR/../test/fancy.min.css $PDF2HTMLEX_DATDIR + +# Do the tests +# +echo "" +echo "-------------------------------------------------------" +echo "running testOutput tests (simple non-browser tests)" +echo "-------------------------------------------------------" +echo "" + +export LAST_TEST_NAME="unknown" +# +runPdf2htmlEX '1-page.pdf' --version + +runPdf2htmlEX '1-page.pdf' +hasExpectedFiles '1-page.html' + +runPdf2htmlEX '2-pages.pdf' +hasExpectedFiles '2-pages.html' + +runPdf2htmlEX '1-page.pdf' 'foo.html' +hasExpectedFiles 'foo.html' + +runPdf2htmlEX '2-pages.pdf' 'foo.html' +hasExpectedFiles 'foo.html' + +runPdf2htmlEX '1-page.pdf' '--split-pages=1' +hasExpectedFiles '1-page.html 1-page1.page' + +runPdf2htmlEX '3-pages.pdf' '--split-pages=1' +hasExpectedFiles '3-pages.html 3-pages1.page 3-pages2.page 3-pages3.page' + +runPdf2htmlEX '1-page.pdf' '--split-pages=1 --page-filename=foo.xyz' +hasExpectedFiles '1-page.html foo1.xyz' + +runPdf2htmlEX '3-pages.pdf' '--split-pages=1 --page-filename=foo.xyz' +hasExpectedFiles '3-pages.html foo1.xyz foo2.xyz foo3.xyz' + +runPdf2htmlEX '3-pages.pdf' '--split-pages=1 --page-filename=fo%do.xyz' +hasExpectedFiles '3-pages.html fo1o.xyz fo2o.xyz fo3o.xyz' + +runPdf2htmlEX '3-pages.pdf' '--split-pages=1 --page-filename=fo%03do.xyz' +hasExpectedFiles '3-pages.html fo001o.xyz fo002o.xyz fo003o.xyz' + +runPdf2htmlEX '3-pages.pdf' '--split-pages=1 --page-filename=f%do%do.xyz' +hasExpectedFiles '3-pages.html f1o%do.xyz f2o%do.xyz f3o%do.xyz' + +runPdf2htmlEX '3-pages.pdf' '--split-pages=1 --page-filename=f%soo.xyz' +hasExpectedFiles '3-pages.html f%soo1.xyz f%soo2.xyz f%soo3.xyz' + +runPdf2htmlEX '3-pages.pdf' '--split-pages=1 --page-filename=f%poo.xyz' +hasExpectedFiles '3-pages.html f%poo1.xyz f%poo2.xyz f%poo3.xyz' + +runPdf2htmlEX '3-pages.pdf' '--split-pages=1 --page-filename=f%noo.xyz' +hasExpectedFiles '3-pages.html f%noo1.xyz f%noo2.xyz f%noo3.xyz' + +runPdf2htmlEX '3-pages.pdf' '--split-pages=1 --page-filename=f%%oo.xyz' +hasExpectedFiles '3-pages.html f%%oo1.xyz f%%oo2.xyz f%%oo3.xyz' + +runPdf2htmlEX '3-pages.pdf' '--split-pages=1 --page-filename=f%%o%do.xyz' +hasExpectedFiles '3-pages.html f%%o1o.xyz f%%o2o.xyz f%%o3o.xyz' + +runPdf2htmlEX '3-pages.pdf' '--split-pages=1 --page-filename=fo%do%%.xyz' +hasExpectedFiles '3-pages.html fo1o%%.xyz fo2o%%.xyz fo3o%%.xyz' + +runPdf2htmlEX '3-pages.pdf' '--split-pages=1 --page-filename=f%02%doo.xyz' +hasExpectedFiles '3-pages.html f%021oo.xyz f%022oo.xyz f%023oo.xyz' + +runPdf2htmlEX '1-page.pdf' '--split-pages=1 --page-filename=foo' +hasExpectedFiles '1-page.html foo1' + +runPdf2htmlEX '2-pages.pdf' 'foo%d.html' +hasExpectedFiles 'foo%d.html' + +runPdf2htmlEX '2-pages.pdf' 'foo%p.html' +hasExpectedFiles 'foo%p.html' + +runPdf2htmlEX '2-pages.pdf' 'foo%n.html' +hasExpectedFiles 'foo%n.html' + +runPdf2htmlEX '2-pages.pdf' 'foo%%.html' +hasExpectedFiles 'foo%%.html' + +runPdf2htmlEX 'issue501' '--split-pages=1 --embed-css=0' +hasExpectedFiles '' + +# Let the user know what the testOutputResults were +# +echo "" +echo "-------------------------------------------------------" +echo "testOutput results:" +echo "-------------------------------------------------------" +cat testOutputResults +echo "-------------------------------------------------------" +echo "" + +# fail the script IF we find the work 'FAILURE' in the testOutputResults +# +if grep FAILURE testOutputResults ; then + exit 1 +fi diff --git a/pdf2htmlEX/test/test_local_browser.py b/pdf2htmlEX/test/test_local_browser.py index 7c67a3d..62a3657 100755 --- a/pdf2htmlEX/test/test_local_browser.py +++ b/pdf2htmlEX/test/test_local_browser.py @@ -43,4 +43,7 @@ class test_local_browser(BrowserTests, unittest.TestCase): self.browser.save_screenshot(png_file) if __name__ == '__main__': - unittest.main() + # turn warnings OFF to prevent 'ResourceWarning: unclosed