diff --git a/buildScripts/Readme.md b/buildScripts/Readme.md index ba3f775..144c71d 100644 --- a/buildScripts/Readme.md +++ b/buildScripts/Readme.md @@ -4,7 +4,7 @@ Because of its intimate use of *specific* versions of both Poppler and FontForge, cleanly building `pdf2htmlEX` is rather more complex than normal. -The (bash) scripts in this directory help automate this mutli-stage +The (shell) scripts in this directory help automate this mutli-stage process. For all but the most experienced programmers, we *strongly* encourage you @@ -111,12 +111,12 @@ However, to get the matched versions of Poppler and FontForge and then compile them statically, *our* build process becomes much more complex than a "simple", `configure, make, make install` cycle. -Hence this directory has a large number of bash scripts each of which +Hence this directory has a large number of shell scripts each of which automate one simple step in the overall our build process. ## The gory details ... -The bash scripts in this directory automate the download, build, install, +The shell scripts in this directory automate the download, build, install, test and upload steps required to provide a complete build/test/release cycle of `pdf2htmlEX`. @@ -263,7 +263,7 @@ Typically, most users, will run one of the following "top-level" scripts: for a standard build of `pdf2htmlEX`. Changes in this script effect *all* of the other build scripts. -- **`reSourceVersionEnvs`**: This bash script is automatically generated +- **`reSourceVersionEnvs`**: This shell script is automatically generated by the build scripts as they are run. It records the values of all important environment variables required by the buildScripts. It is typcically `source`d by each script before it preforms any actions. @@ -272,24 +272,24 @@ Typically, most users, will run one of the following "top-level" scripts: console. This script is used by the top-level scripts to ensure the current environment variables are listed before each build. -- **`uploadGitHubReleaseDSL`**: A collection of bash functions used by the +- **`uploadGitHubReleaseDSL`**: A collection of shell functions used by the `uploadGitHubRelease` script to automate the upload of release artefacts. - **`uploadGitHubReleaseMessage`**: The contents of this *text* file is used by the `uploadGitHubRelease` script as the contents of the release message, as visible to the user, for the 'continuous' release section. -- **`dockerFunctions`**: A collection of bash functions used by the +- **`dockerFunctions`**: A collection of shell functions used by the `uploadDockerImage` script to automate the upload of the Docker images to Docker Hub. -- **`listFilesByChangeTime`**: A simple bash script which lists the files +- **`listFilesByChangeTime`**: A simple shell script which lists the files in the buildScripts directory by most recently changed files first. - **`Readme.md`**: This read me file. ## Yet more details? -The various bash script files are meant to be fairly readable. They +The various shell script files are meant to be fairly readable. They contain additional comments about what each step is meant to be doing. diff --git a/buildScripts/buildFontforge b/buildScripts/buildFontforge index 26cf38a..e58d495 100755 --- a/buildScripts/buildFontforge +++ b/buildScripts/buildFontforge @@ -1,8 +1,9 @@ -#!/bin/bash +#!/bin/sh -# This bash script build FontForge +# This shell script build FontForge -source buildScripts/reSourceVersionEnvs +# source buildScripts/reSourceVersionEnvs +. buildScripts/reSourceVersionEnvs echo "" echo "-------------------------------------------------------------------" diff --git a/buildScripts/buildInstallLocally b/buildScripts/buildInstallLocally index bd7a49c..000dd6b 100755 --- a/buildScripts/buildInstallLocally +++ b/buildScripts/buildInstallLocally @@ -1,6 +1,6 @@ -#!/bin/bash +#!/bin/sh -# This bash script builds the complete pdf2htmlEX application LOCALLY +# This shell script builds the complete pdf2htmlEX application LOCALLY # (It does not create the AppImage or Docker images) # Adjust the following two environment variables to suit your needs diff --git a/buildScripts/buildInstallLocallyAlpine b/buildScripts/buildInstallLocallyAlpine new file mode 100755 index 0000000..8fe6d2f --- /dev/null +++ b/buildScripts/buildInstallLocallyAlpine @@ -0,0 +1,57 @@ +#!/bin/sh + +# This shell script builds the complete pdf2htmlEX application LOCALLY +# (It does not create the AppImage or Docker images) + +# Adjust the following two environment variables to suit your needs +# +export UNATTENDED="--assume-yes" +export MAKE_PARALLEL="-j $(nproc)" + +# choose one of the following... +# +# export PDF2HTMLEX_BRANCH="<>" +export PDF2HTMLEX_BRANCH="$(git rev-parse --abbrev-ref HEAD)" + +# The following environment variable determines where the poppler, +# poppler-data, fontforge and pdf2htmlEX packages are installed. +# CHANGE IT TO SUIT YOUR NEEDS: +# +export PDF2HTMLEX_PREFIX=/usr/local + +# Ensure all Apt packages are installed with no user interaction +# +export DEBIAN_FRONTEND=noninteractive + +set -ev + +################ +# do the build + +./buildScripts/versionEnvs + +./buildScripts/reportEnvs + +./buildScripts/getBuildToolsAlpine + +./buildScripts/getDevLibrariesAlpine + +./buildScripts/getPoppler + +./buildScripts/buildPoppler + +./buildScripts/getFontforge + +./buildScripts/buildFontforge + +./buildScripts/buildPdf2htmlEX + +./buildScripts/installPdf2htmlEX + +#./buildScripts/createAppImage + +#./buildScripts/createDockerUbuntuImageFromDeb + +#./buildScripts/uploadGitHubRelease + +#./buildScripts/uploadDockerImage diff --git a/buildScripts/buildPdf2htmlEX b/buildScripts/buildPdf2htmlEX index 5a048bb..daa62a3 100755 --- a/buildScripts/buildPdf2htmlEX +++ b/buildScripts/buildPdf2htmlEX @@ -1,6 +1,6 @@ -#!/bin/bash +#!/bin/sh -# This bash script builds pdf2htmlEX +# This shell script builds pdf2htmlEX echo "" echo "-------------------------------------------------------------------" @@ -12,6 +12,7 @@ echo "" set -ev cd pdf2htmlEX +rm -rf build mkdir build cd build cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$PDF2HTMLEX_PREFIX .. diff --git a/buildScripts/buildPoppler b/buildScripts/buildPoppler index 3253c75..4583ccb 100755 --- a/buildScripts/buildPoppler +++ b/buildScripts/buildPoppler @@ -1,6 +1,6 @@ -#!/bin/bash +#!/bin/sh -# This bash script builds the latest poppler +# This shell script builds the latest poppler echo "" echo "-------------------------------------------------------------------" diff --git a/buildScripts/createAppImage b/buildScripts/createAppImage index 8c33d52..c2e2fa7 100755 --- a/buildScripts/createAppImage +++ b/buildScripts/createAppImage @@ -1,8 +1,9 @@ -#!/bin/bash +#!/bin/sh -# This bash script creates an AppImage for pdf2htmlEX +# This shell script creates an AppImage for pdf2htmlEX -source ./buildScripts/reSourceVersionEnvs +# source ./buildScripts/reSourceVersionEnvs +. ./buildScripts/reSourceVersionEnvs echo "" echo "-------------------------------------------------------------------" diff --git a/buildScripts/createDebianPackage b/buildScripts/createDebianPackage index 1dbff39..64b0186 100755 --- a/buildScripts/createDebianPackage +++ b/buildScripts/createDebianPackage @@ -1,8 +1,9 @@ -#!/bin/bash +#!/bin/sh -# This bash script creates a (binary) Debian Package Archive for pdf2htmlEX +# This shell script creates a (binary) Debian Package Archive for pdf2htmlEX -source ./buildScripts/reSourceVersionEnvs +# source ./buildScripts/reSourceVersionEnvs +. ./buildScripts/reSourceVersionEnvs echo "" echo "-------------------------------------------------------------------" diff --git a/buildScripts/createDockerAlpineImageInside b/buildScripts/createDockerAlpineImageInside new file mode 100755 index 0000000..c2ba067 --- /dev/null +++ b/buildScripts/createDockerAlpineImageInside @@ -0,0 +1,37 @@ +#!/bin/sh + +# This shell script creates a docker image from an existing pdf2htmlEX +# +# This is the part which can run *inside* a docker container. + +echo "" +echo "-------------------------------------------------------------------" +echo "CREATING pdf2htmlEX Docker Image (run inside a docker container)" +echo " (based on Alpine linux $BUILD_DIST) +echo "-------------------------------------------------------------------" +echo "" + +# Collect everything that will be needed... + +# source buildScripts/reSourceVersionEnvs +. buildScripts/reSourceVersionEnvs + +# source buildScripts/dockerFunctions +. buildScripts/dockerFunctions + +set -ev + +mkdir -p imageBuild/dockerDir + +cd pdf2htmlEX/build + +sudo rm -rf install_manifest.txt + +make install DESTDIR=../../imageBuild/dockerDir + +cd ../../poppler-data + +make install \ + prefix=$PDF2HTMLEX_PREFIX \ + datadir=$PDF2HTMLEX_PREFIX/share/pdf2htmlEX \ + DESTDIR=../imageBuild/dockerDir diff --git a/buildScripts/createDockerAlpineImageOutside b/buildScripts/createDockerAlpineImageOutside new file mode 100755 index 0000000..30fe2f1 --- /dev/null +++ b/buildScripts/createDockerAlpineImageOutside @@ -0,0 +1,77 @@ +#!/bin/sh + +# This shell script creates a docker image from an existing pdf2htmlEX +# +# This is the part which must be run *outside* of any docker container. + +echo "" +echo "-------------------------------------------------------------------" +echo "CREATING pdf2htmlEX Docker Image (run outside any docker container)" +echo " (based on Alpine linux $BUILD_DIST) +echo "-------------------------------------------------------------------" +echo "" + +# Collect everything that will be needed... + +# source buildScripts/reSourceVersionEnvs +. buildScripts/reSourceVersionEnvs + +# source buildScripts/dockerFunctions +. buildScripts/dockerFunctions + +set -ev + +mkdir -p imageBuild/dockerDir + +cd imageBuild/dockerDir + +if [ -z "$DOCKER_FROM" ]; then + echo "" + read -p "Enter the docker image for the 'from' base: " DOCKER_FROM + echo "" + if [ -z "$DOCKER_FROM" ]; then + echo "DOCKER_FROM not set... so we can not build the docker image" + exit 1 + fi +fi + +if [ -z "$DOCKER_USERNAME" ]; then + echo "" + read -p "Enter a docker username: " DOCKER_USERNAME + echo "" + if [ -z "$DOCKER_USERNAME" ]; then + echo "DOCKER_USERNAME not set... so we can not build the docker image" + exit 1 + fi +fi + +export DOCKER_NAME="$DOCKER_USERNAME/pdf2htmlex:$PDF2HTMLEX_BRANCH-$BUILD_TIME" + +echo "export DOCKER_FROM=\"$DOCKER_FROM\"" >> ../../buildScripts/reSourceVersionEnvs +echo "export DOCKER_USERNAME=\"$DOCKER_USERNAME\"" >> ../../buildScripts/reSourceVersionEnvs +echo "export DOCKER_NAME=\"$DOCKER_NAME\"" >> ../../buildScripts/reSourceVersionEnvs + +delete_blacklisted + +cat > Dockerfile <> buildScripts/reSourceV set -ev -# This bash script runs the pdf2htmlEX tests +# This shell script runs the pdf2htmlEX tests cd pdf2htmlEX/test @@ -21,7 +22,7 @@ cd pdf2htmlEX/test # ./installAutomaticTestSoftware -./runLocalTestsBash +./runLocalTestsShell ./runLocalBrowserTests diff --git a/buildScripts/travisLinuxDoItAll b/buildScripts/travisLinuxDoItAll index 063d5b3..6ac8028 100755 --- a/buildScripts/travisLinuxDoItAll +++ b/buildScripts/travisLinuxDoItAll @@ -1,6 +1,6 @@ -#!/bin/bash +#!/bin/sh -# This bash script builds everyting on an TravisCI Linux (Ubunutu) worker +# This shell script builds everyting on an TravisCI Linux (Ubunutu) worker set -ev diff --git a/buildScripts/uploadDockerImage b/buildScripts/uploadDockerImage index 13fe66f..48cb0ec 100755 --- a/buildScripts/uploadDockerImage +++ b/buildScripts/uploadDockerImage @@ -1,6 +1,6 @@ -#!/bin/bash +#!/bin/sh -# This bash script uploads the pdf2htmlEX docker image to docker hub +# This shell script uploads the pdf2htmlEX docker image to docker hub # # We EXPECT the following environment variables to be set: # DOCKER_USERNAME @@ -9,7 +9,8 @@ # DOCKER_PASSWORD (if not set you will be asked for your password) # -source ./buildScripts/reSourceVersionEnvs +# source ./buildScripts/reSourceVersionEnvs +. ./buildScripts/reSourceVersionEnvs echo "" echo "-------------------------------------------------------------------" diff --git a/buildScripts/uploadGitHubRelease b/buildScripts/uploadGitHubRelease index ad4d690..ffde585 100755 --- a/buildScripts/uploadGitHubRelease +++ b/buildScripts/uploadGitHubRelease @@ -1,6 +1,6 @@ -#!/bin/bash +#!/bin/sh -# This bash script uploads the pdf2htmlEX release artefacts +# This shell script uploads the pdf2htmlEX release artefacts # # We EXPECT the following environment variables to be set: # GITHUB_TOKEN @@ -10,9 +10,11 @@ # DOCKER_PASSWORD (if not set you will be asked for your password) # -source ./buildScripts/reSourceVersionEnvs -cp ./buildScripts/reSourceVersionEnvs imageBuild/buildInfo.sh -source ./buildScripts/uploadGitHubReleaseDSL +# source ./buildScripts/reSourceVersionEnvs +. ./buildScripts/reSourceVersionEnvs +cp ./buildScripts/reSourceVersionEnvs imageBuild/buildInfo.sh +. ./buildScripts/uploadGitHubReleaseDSL +# source ./buildScripts/uploadGitHubReleaseDSL echo "" echo "-------------------------------------------------------------------" diff --git a/buildScripts/uploadGitHubReleaseDSL b/buildScripts/uploadGitHubReleaseDSL index 3a0207a..eb226d6 100755 --- a/buildScripts/uploadGitHubReleaseDSL +++ b/buildScripts/uploadGitHubReleaseDSL @@ -1,8 +1,8 @@ -#!/usr/bin/env bash +#!/usr/bin/env sh -# This bash script is a collection of functions to interact with the GitHub -# release and git/tag APIs. It essentially provides a bash based DSL for -# uploading GitHub release artefacts. +# This shell script is a collection of functions to interact with the GitHub +# release and git/tag APIs. It essentially provides a shell script based DSL +# for uploading GitHub release artefacts. # NOTE-1: These functions make use of your ".netrc" file to authenticate # with GitHub. In order to use these functions, make sure you have **both** @@ -171,7 +171,7 @@ function uploadAnAsset { echo "" } -# The above has been heavily modified into a collection of bash functions +# The above has been heavily modified into a collection of shell functions # by Stephen Gaito working on the pdf2htmlEX project. # # It has been based upon the code in: diff --git a/buildScripts/uploadImages b/buildScripts/uploadImages index 7adbe9c..28760a0 100755 --- a/buildScripts/uploadImages +++ b/buildScripts/uploadImages @@ -1,6 +1,6 @@ -#!/bin/bash +#!/bin/sh -# This bash script uploads already existing pdf2htmlEX AppImage and Docker +# This shell script uploads already existing pdf2htmlEX AppImage and Docker # Images ################ diff --git a/buildScripts/versionEnvs b/buildScripts/versionEnvs index d8c1da6..637a3b7 100755 --- a/buildScripts/versionEnvs +++ b/buildScripts/versionEnvs @@ -1,6 +1,6 @@ -#!/bin/bash +#!/bin/sh -# This bash script exports environment variables for the latest software +# This shell script exports environment variables for the latest software # versions # see: https://poppler.freedesktop.org/releases.html @@ -42,11 +42,18 @@ if [ -z "$PDF2HTMLEX_BRANCH" ]; then fi fi -cat /etc/lsb-release - -source /etc/lsb-release - -export BUILD_DIST=$DISTRIB_CODENAME +if test -r /etc/lsb-release ; then + cat /etc/lsb-release + # source /etc/lsb-release + . /etc/lsb-release + export BUILD_DIST=$DISTRIB_CODENAME +elif test -r /etc/alpine-release ; then + cat /etc/alpine-release + export BUILD_DIST=$(cat /etc/alpine-release) +else + echo "FAILURE: could not determine release" + exit -1 +fi export BUILD_DATE="$(date +%Y%m%d)" diff --git a/pdf2htmlEX/CMakeLists.txt b/pdf2htmlEX/CMakeLists.txt index ab3c61a..218c54d 100644 --- a/pdf2htmlEX/CMakeLists.txt +++ b/pdf2htmlEX/CMakeLists.txt @@ -79,9 +79,18 @@ set(FONTFORGE_LIBRARIES ${FONTFORGE_LIBRARIES} ${CMAKE_SOURCE_DIR}/../fontforge/build/lib/libfontforge.a ) +# If we are using Alpine Linux then we need to add -lintl +# +if (EXISTS /usr/lib/libintl.so ) + set(LIB_INTL_LIBRARIES -lintl ) +else () + set(LIB_INTL_LIBRARIES "" ) +endif() + set(PDF2HTMLEX_LIBS ${PDF2HTMLEX_LIBS} ${POPPLER_LIBRARIES} ${FONTFORGE_LIBRARIES} + ${LIB_INTL_LIBRARIES} ${CAIRO_LIBRARIES} -ljpeg -lpng diff --git a/pdf2htmlEX/test/README.md b/pdf2htmlEX/test/README.md index 7b8f708..24a0209 100644 --- a/pdf2htmlEX/test/README.md +++ b/pdf2htmlEX/test/README.md @@ -19,7 +19,7 @@ and needs to be fixed. Rerun `runLocalBrowserTests` and use the ## Running tests -There are three bash scripts which automate the running of a given +There are three shell scripts which automate the running of a given collection of tests: 1. **runLocalTests** runs a simple collection of tests which do not diff --git a/pdf2htmlEX/test/installAutomaticTestSoftware b/pdf2htmlEX/test/installAutomaticTestSoftware index 2b53ad2..d1b73dd 100755 --- a/pdf2htmlEX/test/installAutomaticTestSoftware +++ b/pdf2htmlEX/test/installAutomaticTestSoftware @@ -1,8 +1,8 @@ -#!/bin/bash +#!/bin/sh set -ev -# This bash script installs all local software required to run the +# This shell script installs all local software required to run the # pdf2htmlEX tests export DEBIAN_FRONTEND=noninteractive diff --git a/pdf2htmlEX/test/installManualTestSoftware b/pdf2htmlEX/test/installManualTestSoftware index d7c8ab3..5dc3e8a 100755 --- a/pdf2htmlEX/test/installManualTestSoftware +++ b/pdf2htmlEX/test/installManualTestSoftware @@ -1,6 +1,6 @@ -#!/bin/bash +#!/bin/sh -# This bash script installs all local software required to run the +# This shell script installs all local software required to run the # pdf2htmlEX tests export DEBIAN_FRONTEND=noninteractive diff --git a/pdf2htmlEX/test/produceHtmlForBrowserTests b/pdf2htmlEX/test/produceHtmlForBrowserTests index 6225aab..c0979a6 100755 --- a/pdf2htmlEX/test/produceHtmlForBrowserTests +++ b/pdf2htmlEX/test/produceHtmlForBrowserTests @@ -1,6 +1,6 @@ -#!/bin/bash +#!/bin/sh -# This bash script walks through the browser_tests directory running +# This shell script walks through the browser_tests directory running # pdf2htmlEX on each *.pdf file. # This is how we run pdf2htmlEX on a particular file, and arguments. diff --git a/pdf2htmlEX/test/runLocalBrowserTests b/pdf2htmlEX/test/runLocalBrowserTests index f65af7b..f27cae2 100755 --- a/pdf2htmlEX/test/runLocalBrowserTests +++ b/pdf2htmlEX/test/runLocalBrowserTests @@ -1,6 +1,6 @@ -#!/bin/bash +#!/bin/sh -# This bash script runs the local browser tests +# This shell script runs the local browser tests # We start by running a virtual frame buffer as display 99.0 # diff --git a/pdf2htmlEX/test/runLocalTestsBash b/pdf2htmlEX/test/runLocalTestsBash deleted file mode 100755 index f3bf575..0000000 --- a/pdf2htmlEX/test/runLocalTestsBash +++ /dev/null @@ -1,7 +0,0 @@ -#!/bin/bash - -set -ev - -# This bash script runs the (simple non-browser) tests - -./testOutput diff --git a/pdf2htmlEX/test/runLocalTestsPython b/pdf2htmlEX/test/runLocalTestsPython index f8db9c7..8f0fa85 100755 --- a/pdf2htmlEX/test/runLocalTestsPython +++ b/pdf2htmlEX/test/runLocalTestsPython @@ -1,7 +1,7 @@ -#!/bin/bash +#!/bin/sh set -ev -# This bash script runs the (simple non-browser) tests +# This shell script runs the (simple non-browser) tests python3 test_output.py diff --git a/pdf2htmlEX/test/runLocalTestsShell b/pdf2htmlEX/test/runLocalTestsShell new file mode 100755 index 0000000..692245a --- /dev/null +++ b/pdf2htmlEX/test/runLocalTestsShell @@ -0,0 +1,7 @@ +#!/bin/sh + +set -ev + +# This shell script runs the (simple non-browser) tests + +./testOutput diff --git a/pdf2htmlEX/test/testOutput b/pdf2htmlEX/test/testOutput index c18ebdb..8e11fd3 100755 --- a/pdf2htmlEX/test/testOutput +++ b/pdf2htmlEX/test/testOutput @@ -1,9 +1,9 @@ -#!/bin/bash +#!/bin/sh -# This bash script checks that pdf2htmlEX does not crash, and produces +# This shell script checks that pdf2htmlEX does not crash, and produces # correct files. Do not check the content of the files. -# We use a bash script to implement the python3 test_output.py so that we +# We use a shell script to implement the python3 test_output.py so that we # do not need to install any extra packages to test this functionality. # This is how we run pdf2htmlEX on a particular file, and arguments.