first attempt at integration of build on Alpine Linx

This commit is contained in:
Stephen Gaito 2020-06-20 09:11:30 +00:00
parent 32f37576d5
commit f1abed5a1f
38 changed files with 348 additions and 95 deletions

View File

@ -4,7 +4,7 @@ Because of its intimate use of *specific* versions of both Poppler and
FontForge, cleanly building `pdf2htmlEX` is rather more complex than
normal.
The (bash) scripts in this directory help automate this mutli-stage
The (shell) scripts in this directory help automate this mutli-stage
process.
For all but the most experienced programmers, we *strongly* encourage you
@ -111,12 +111,12 @@ However, to get the matched versions of Poppler and FontForge and then
compile them statically, *our* build process becomes much more complex
than a "simple", `configure, make, make install` cycle.
Hence this directory has a large number of bash scripts each of which
Hence this directory has a large number of shell scripts each of which
automate one simple step in the overall our build process.
## The gory details ...
The bash scripts in this directory automate the download, build, install,
The shell scripts in this directory automate the download, build, install,
test and upload steps required to provide a complete build/test/release
cycle of `pdf2htmlEX`.
@ -263,7 +263,7 @@ Typically, most users, will run one of the following "top-level" scripts:
for a standard build of `pdf2htmlEX`. Changes in this script effect
*all* of the other build scripts.
- **`reSourceVersionEnvs`**: This bash script is automatically generated
- **`reSourceVersionEnvs`**: This shell script is automatically generated
by the build scripts as they are run. It records the values of all
important environment variables required by the buildScripts. It is
typcically `source`d by each script before it preforms any actions.
@ -272,24 +272,24 @@ Typically, most users, will run one of the following "top-level" scripts:
console. This script is used by the top-level scripts to ensure the
current environment variables are listed before each build.
- **`uploadGitHubReleaseDSL`**: A collection of bash functions used by the
- **`uploadGitHubReleaseDSL`**: A collection of shell functions used by the
`uploadGitHubRelease` script to automate the upload of release artefacts.
- **`uploadGitHubReleaseMessage`**: The contents of this *text* file is
used by the `uploadGitHubRelease` script as the contents of the release
message, as visible to the user, for the 'continuous' release section.
- **`dockerFunctions`**: A collection of bash functions used by the
- **`dockerFunctions`**: A collection of shell functions used by the
`uploadDockerImage` script to automate the upload of the Docker images
to Docker Hub.
- **`listFilesByChangeTime`**: A simple bash script which lists the files
- **`listFilesByChangeTime`**: A simple shell script which lists the files
in the buildScripts directory by most recently changed files first.
- **`Readme.md`**: This read me file.
## Yet more details?
The various bash script files are meant to be fairly readable. They
The various shell script files are meant to be fairly readable. They
contain additional comments about what each step is meant to be doing.

View File

@ -1,8 +1,9 @@
#!/bin/bash
#!/bin/sh
# This bash script build FontForge
# This shell script build FontForge
source buildScripts/reSourceVersionEnvs
# source buildScripts/reSourceVersionEnvs
. buildScripts/reSourceVersionEnvs
echo ""
echo "-------------------------------------------------------------------"

View File

@ -1,6 +1,6 @@
#!/bin/bash
#!/bin/sh
# This bash script builds the complete pdf2htmlEX application LOCALLY
# This shell script builds the complete pdf2htmlEX application LOCALLY
# (It does not create the AppImage or Docker images)
# Adjust the following two environment variables to suit your needs

View File

@ -0,0 +1,57 @@
#!/bin/sh
# This shell script builds the complete pdf2htmlEX application LOCALLY
# (It does not create the AppImage or Docker images)
# Adjust the following two environment variables to suit your needs
#
export UNATTENDED="--assume-yes"
export MAKE_PARALLEL="-j $(nproc)"
# choose one of the following...
#
# export PDF2HTMLEX_BRANCH="<<YourTagHereWithNoSpaces>>"
export PDF2HTMLEX_BRANCH="$(git rev-parse --abbrev-ref HEAD)"
# The following environment variable determines where the poppler,
# poppler-data, fontforge and pdf2htmlEX packages are installed.
# CHANGE IT TO SUIT YOUR NEEDS:
#
export PDF2HTMLEX_PREFIX=/usr/local
# Ensure all Apt packages are installed with no user interaction
#
export DEBIAN_FRONTEND=noninteractive
set -ev
################
# do the build
./buildScripts/versionEnvs
./buildScripts/reportEnvs
./buildScripts/getBuildToolsAlpine
./buildScripts/getDevLibrariesAlpine
./buildScripts/getPoppler
./buildScripts/buildPoppler
./buildScripts/getFontforge
./buildScripts/buildFontforge
./buildScripts/buildPdf2htmlEX
./buildScripts/installPdf2htmlEX
#./buildScripts/createAppImage
#./buildScripts/createDockerUbuntuImageFromDeb
#./buildScripts/uploadGitHubRelease
#./buildScripts/uploadDockerImage

View File

@ -1,6 +1,6 @@
#!/bin/bash
#!/bin/sh
# This bash script builds pdf2htmlEX
# This shell script builds pdf2htmlEX
echo ""
echo "-------------------------------------------------------------------"
@ -12,6 +12,7 @@ echo ""
set -ev
cd pdf2htmlEX
rm -rf build
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$PDF2HTMLEX_PREFIX ..

View File

@ -1,6 +1,6 @@
#!/bin/bash
#!/bin/sh
# This bash script builds the latest poppler
# This shell script builds the latest poppler
echo ""
echo "-------------------------------------------------------------------"

View File

@ -1,8 +1,9 @@
#!/bin/bash
#!/bin/sh
# This bash script creates an AppImage for pdf2htmlEX
# This shell script creates an AppImage for pdf2htmlEX
source ./buildScripts/reSourceVersionEnvs
# source ./buildScripts/reSourceVersionEnvs
. ./buildScripts/reSourceVersionEnvs
echo ""
echo "-------------------------------------------------------------------"

View File

@ -1,8 +1,9 @@
#!/bin/bash
#!/bin/sh
# This bash script creates a (binary) Debian Package Archive for pdf2htmlEX
# This shell script creates a (binary) Debian Package Archive for pdf2htmlEX
source ./buildScripts/reSourceVersionEnvs
# source ./buildScripts/reSourceVersionEnvs
. ./buildScripts/reSourceVersionEnvs
echo ""
echo "-------------------------------------------------------------------"

View File

@ -0,0 +1,37 @@
#!/bin/sh
# This shell script creates a docker image from an existing pdf2htmlEX
#
# This is the part which can run *inside* a docker container.
echo ""
echo "-------------------------------------------------------------------"
echo "CREATING pdf2htmlEX Docker Image (run inside a docker container)"
echo " (based on Alpine linux $BUILD_DIST)
echo "-------------------------------------------------------------------"
echo ""
# Collect everything that will be needed...
# source buildScripts/reSourceVersionEnvs
. buildScripts/reSourceVersionEnvs
# source buildScripts/dockerFunctions
. buildScripts/dockerFunctions
set -ev
mkdir -p imageBuild/dockerDir
cd pdf2htmlEX/build
sudo rm -rf install_manifest.txt
make install DESTDIR=../../imageBuild/dockerDir
cd ../../poppler-data
make install \
prefix=$PDF2HTMLEX_PREFIX \
datadir=$PDF2HTMLEX_PREFIX/share/pdf2htmlEX \
DESTDIR=../imageBuild/dockerDir

View File

@ -0,0 +1,77 @@
#!/bin/sh
# This shell script creates a docker image from an existing pdf2htmlEX
#
# This is the part which must be run *outside* of any docker container.
echo ""
echo "-------------------------------------------------------------------"
echo "CREATING pdf2htmlEX Docker Image (run outside any docker container)"
echo " (based on Alpine linux $BUILD_DIST)
echo "-------------------------------------------------------------------"
echo ""
# Collect everything that will be needed...
# source buildScripts/reSourceVersionEnvs
. buildScripts/reSourceVersionEnvs
# source buildScripts/dockerFunctions
. buildScripts/dockerFunctions
set -ev
mkdir -p imageBuild/dockerDir
cd imageBuild/dockerDir
if [ -z "$DOCKER_FROM" ]; then
echo ""
read -p "Enter the docker image for the 'from' base: " DOCKER_FROM
echo ""
if [ -z "$DOCKER_FROM" ]; then
echo "DOCKER_FROM not set... so we can not build the docker image"
exit 1
fi
fi
if [ -z "$DOCKER_USERNAME" ]; then
echo ""
read -p "Enter a docker username: " DOCKER_USERNAME
echo ""
if [ -z "$DOCKER_USERNAME" ]; then
echo "DOCKER_USERNAME not set... so we can not build the docker image"
exit 1
fi
fi
export DOCKER_NAME="$DOCKER_USERNAME/pdf2htmlex:$PDF2HTMLEX_BRANCH-$BUILD_TIME"
echo "export DOCKER_FROM=\"$DOCKER_FROM\"" >> ../../buildScripts/reSourceVersionEnvs
echo "export DOCKER_USERNAME=\"$DOCKER_USERNAME\"" >> ../../buildScripts/reSourceVersionEnvs
echo "export DOCKER_NAME=\"$DOCKER_NAME\"" >> ../../buildScripts/reSourceVersionEnvs
delete_blacklisted
cat > Dockerfile <<DOCKERFILE_HERE_DOC
FROM $DOCKER_FROM
COPY ./ /
RUN apk update && \
apk add \
gettext \
glib \
freetype \
fontconfig \
cairo \
libpng \
libjpeg-turbo \
libxml2
ENTRYPOINT ["$PDF2HTMLEX_PREFIX/bin/pdf2htmlEX"]
DOCKERFILE_HERE_DOC
cd ..
docker build -t $DOCKER_NAME dockerDir

View File

@ -1,6 +1,6 @@
#!/bin/bash
#!/bin/sh
# This bash script creates a docker image from an existing pdf2htmlEX
# This shell script creates a docker image from an existing pdf2htmlEX
#
# This is the part which must be run *outside* of any docker container.
@ -12,9 +12,11 @@ echo ""
# Collect everything that will be needed...
source buildScripts/reSourceVersionEnvs
# source buildScripts/reSourceVersionEnvs
. buildScripts/reSourceVersionEnvs
source buildScripts/dockerFunctions
# source buildScripts/dockerFunctions
. buildScripts/dockerFunctions
set -ev

View File

@ -1,8 +1,8 @@
#!/bin/bash
#!/bin/sh
set -ev
# This bash script creates the pdf2htmlEX AppImage and Docker Images
# This shell script creates the pdf2htmlEX AppImage and Docker Images
#################
# do the creation

View File

@ -1,8 +1,8 @@
#!/bin/bash
#!/bin/sh
# This bash script automates getting the required build tools (apt install)
# This shell script automates getting the required build tools (apt install)
# set the bash environment variable 'UNATTENDED' to '--assume-yes' for
# set the shell environment variable 'UNATTENDED' to '--assume-yes' for
# unattended use (for example in the .travis.yml script)
echo ""

View File

@ -0,0 +1,30 @@
#!/bin/sh
# This shell script automates getting the required build tools (apt install)
# set the shell environment variable 'UNATTENDED' to '--assume-yes' for
# unattended use (for example in the .travis.yml script)
echo ""
echo "-------------------------------------------------------------------"
echo "INSTALLING Build Tools (using APK / Alpine)"
echo "-------------------------------------------------------------------"
echo ""
set -ev
sudo apk update
sudo apk add \
sudo \
wget \
git \
pkgconfig \
ruby \
cmake \
make \
gcc \
g++ \
gettext \
openjdk8 \
jq

View File

@ -1,9 +1,9 @@
#!/bin/bash
#!/bin/sh
# This bash script automates getting the development libraries required to
# This shell script automates getting the development libraries required to
# build poppler and fontforge
# set the bash environment variable 'UNATTENDED' to '--assume-yes' for
# set the shell environment variable 'UNATTENDED' to '--assume-yes' for
# unattended use (for example in the .travis.yml script)
echo ""

View File

@ -0,0 +1,23 @@
#!/bin/sh
# This shell script automates getting the development libraries required to
# build poppler and fontforge
# set the shell environment variable 'UNATTENDED' to '--assume-yes' for
# unattended use (for example in the .travis.yml script)
echo ""
echo "-------------------------------------------------------------------"
echo "INSTALLING development libraries (using APK on Alpine)"
echo "-------------------------------------------------------------------"
echo ""
set -ev
sudo apk update
sudo apk add \
gettext \
cairo \
libpng-dev \
libjpeg-turbo-dev \
libxml2-dev

View File

@ -1,8 +1,9 @@
#!/bin/bash
#!/bin/sh
# This bash script gets and unpacks the latest fontforge AppImage
# This shell script gets and unpacks the latest fontforge AppImage
source buildScripts/reSourceVersionEnvs
# source buildScripts/reSourceVersionEnvs
. buildScripts/reSourceVersionEnvs
echo ""
echo "-------------------------------------------------------------------"

View File

@ -1,8 +1,9 @@
#!/bin/bash
#!/bin/sh
# This bash script gets and unpacks the latest Poppler source code
# This shell script gets and unpacks the latest Poppler source code
source buildScripts/reSourceVersionEnvs
# source buildScripts/reSourceVersionEnvs
. buildScripts/reSourceVersionEnvs
echo ""
echo "-------------------------------------------------------------------"

View File

@ -1,4 +1,4 @@
#!/bin/bash
#!/bin/sh
echo ""
echo "-------------------------------------------------------------------"

View File

@ -1,4 +1,4 @@
#!/bin/bash
#!/bin/sh
# see:https://stackoverflow.com/a/7448828

View File

@ -1,8 +1,9 @@
#!/bin/bash
#!/bin/sh
# This bash script reports the more important TRAVIS environment variables
# This shell script reports the more important TRAVIS environment variables
source ./buildScripts/reSourceVersionEnvs
# source ./buildScripts/reSourceVersionEnvs
. ./buildScripts/reSourceVersionEnvs
echo "TravisCI env:"
echo " dist: [$TRAVIS_DIST]"

View File

@ -1,6 +1,7 @@
#!/bin/bash
#!/bin/sh
source ./buildScripts/reSourceVersionEnvs
# source ./buildScripts/reSourceVersionEnvs
. ./buildScripts/reSourceVersionEnvs
echo ""
echo "-------------------------------------------------------------------"
@ -13,7 +14,7 @@ echo "export TEST_RESULTS_NAME=\"$TEST_RESULTS_NAME\"" >> buildScripts/reSourceV
set -ev
# This bash script runs the pdf2htmlEX tests
# This shell script runs the pdf2htmlEX tests
cd pdf2htmlEX/test
@ -21,7 +22,7 @@ cd pdf2htmlEX/test
#
./installAutomaticTestSoftware
./runLocalTestsBash
./runLocalTestsShell
./runLocalBrowserTests

View File

@ -1,6 +1,6 @@
#!/bin/bash
#!/bin/sh
# This bash script builds everyting on an TravisCI Linux (Ubunutu) worker
# This shell script builds everyting on an TravisCI Linux (Ubunutu) worker
set -ev

View File

@ -1,6 +1,6 @@
#!/bin/bash
#!/bin/sh
# This bash script uploads the pdf2htmlEX docker image to docker hub
# This shell script uploads the pdf2htmlEX docker image to docker hub
#
# We EXPECT the following environment variables to be set:
# DOCKER_USERNAME
@ -9,7 +9,8 @@
# DOCKER_PASSWORD (if not set you will be asked for your password)
#
source ./buildScripts/reSourceVersionEnvs
# source ./buildScripts/reSourceVersionEnvs
. ./buildScripts/reSourceVersionEnvs
echo ""
echo "-------------------------------------------------------------------"

View File

@ -1,6 +1,6 @@
#!/bin/bash
#!/bin/sh
# This bash script uploads the pdf2htmlEX release artefacts
# This shell script uploads the pdf2htmlEX release artefacts
#
# We EXPECT the following environment variables to be set:
# GITHUB_TOKEN
@ -10,9 +10,11 @@
# DOCKER_PASSWORD (if not set you will be asked for your password)
#
source ./buildScripts/reSourceVersionEnvs
cp ./buildScripts/reSourceVersionEnvs imageBuild/buildInfo.sh
source ./buildScripts/uploadGitHubReleaseDSL
# source ./buildScripts/reSourceVersionEnvs
. ./buildScripts/reSourceVersionEnvs
cp ./buildScripts/reSourceVersionEnvs imageBuild/buildInfo.sh
. ./buildScripts/uploadGitHubReleaseDSL
# source ./buildScripts/uploadGitHubReleaseDSL
echo ""
echo "-------------------------------------------------------------------"

View File

@ -1,8 +1,8 @@
#!/usr/bin/env bash
#!/usr/bin/env sh
# This bash script is a collection of functions to interact with the GitHub
# release and git/tag APIs. It essentially provides a bash based DSL for
# uploading GitHub release artefacts.
# This shell script is a collection of functions to interact with the GitHub
# release and git/tag APIs. It essentially provides a shell script based DSL
# for uploading GitHub release artefacts.
# NOTE-1: These functions make use of your ".netrc" file to authenticate
# with GitHub. In order to use these functions, make sure you have **both**
@ -171,7 +171,7 @@ function uploadAnAsset {
echo ""
}
# The above has been heavily modified into a collection of bash functions
# The above has been heavily modified into a collection of shell functions
# by Stephen Gaito working on the pdf2htmlEX project.
#
# It has been based upon the code in:

View File

@ -1,6 +1,6 @@
#!/bin/bash
#!/bin/sh
# This bash script uploads already existing pdf2htmlEX AppImage and Docker
# This shell script uploads already existing pdf2htmlEX AppImage and Docker
# Images
################

View File

@ -1,6 +1,6 @@
#!/bin/bash
#!/bin/sh
# This bash script exports environment variables for the latest software
# This shell script exports environment variables for the latest software
# versions
# see: https://poppler.freedesktop.org/releases.html
@ -42,11 +42,18 @@ if [ -z "$PDF2HTMLEX_BRANCH" ]; then
fi
fi
cat /etc/lsb-release
source /etc/lsb-release
export BUILD_DIST=$DISTRIB_CODENAME
if test -r /etc/lsb-release ; then
cat /etc/lsb-release
# source /etc/lsb-release
. /etc/lsb-release
export BUILD_DIST=$DISTRIB_CODENAME
elif test -r /etc/alpine-release ; then
cat /etc/alpine-release
export BUILD_DIST=$(cat /etc/alpine-release)
else
echo "FAILURE: could not determine release"
exit -1
fi
export BUILD_DATE="$(date +%Y%m%d)"

View File

@ -79,9 +79,18 @@ set(FONTFORGE_LIBRARIES ${FONTFORGE_LIBRARIES}
${CMAKE_SOURCE_DIR}/../fontforge/build/lib/libfontforge.a
)
# If we are using Alpine Linux then we need to add -lintl
#
if (EXISTS /usr/lib/libintl.so )
set(LIB_INTL_LIBRARIES -lintl )
else ()
set(LIB_INTL_LIBRARIES "" )
endif()
set(PDF2HTMLEX_LIBS ${PDF2HTMLEX_LIBS}
${POPPLER_LIBRARIES}
${FONTFORGE_LIBRARIES}
${LIB_INTL_LIBRARIES}
${CAIRO_LIBRARIES}
-ljpeg
-lpng

View File

@ -19,7 +19,7 @@ and needs to be fixed. Rerun `runLocalBrowserTests` and use the
## Running tests
There are three bash scripts which automate the running of a given
There are three shell scripts which automate the running of a given
collection of tests:
1. **runLocalTests** runs a simple collection of tests which do not

View File

@ -1,8 +1,8 @@
#!/bin/bash
#!/bin/sh
set -ev
# This bash script installs all local software required to run the
# This shell script installs all local software required to run the
# pdf2htmlEX tests
export DEBIAN_FRONTEND=noninteractive

View File

@ -1,6 +1,6 @@
#!/bin/bash
#!/bin/sh
# This bash script installs all local software required to run the
# This shell script installs all local software required to run the
# pdf2htmlEX tests
export DEBIAN_FRONTEND=noninteractive

View File

@ -1,6 +1,6 @@
#!/bin/bash
#!/bin/sh
# This bash script walks through the browser_tests directory running
# This shell script walks through the browser_tests directory running
# pdf2htmlEX on each *.pdf file.
# This is how we run pdf2htmlEX on a particular file, and arguments.

View File

@ -1,6 +1,6 @@
#!/bin/bash
#!/bin/sh
# This bash script runs the local browser tests
# This shell script runs the local browser tests
# We start by running a virtual frame buffer as display 99.0
#

View File

@ -1,7 +0,0 @@
#!/bin/bash
set -ev
# This bash script runs the (simple non-browser) tests
./testOutput

View File

@ -1,7 +1,7 @@
#!/bin/bash
#!/bin/sh
set -ev
# This bash script runs the (simple non-browser) tests
# This shell script runs the (simple non-browser) tests
python3 test_output.py

View File

@ -0,0 +1,7 @@
#!/bin/sh
set -ev
# This shell script runs the (simple non-browser) tests
./testOutput

View File

@ -1,9 +1,9 @@
#!/bin/bash
#!/bin/sh
# This bash script checks that pdf2htmlEX does not crash, and produces
# This shell script checks that pdf2htmlEX does not crash, and produces
# correct files. Do not check the content of the files.
# We use a bash script to implement the python3 test_output.py so that we
# We use a shell script to implement the python3 test_output.py so that we
# do not need to install any extra packages to test this functionality.
# This is how we run pdf2htmlEX on a particular file, and arguments.