[Treepkg-commits] r400 - in trunk: . bin test treepkg treepkg/info
scm-commit@wald.intevation.org
scm-commit at wald.intevation.org
Fri Aug 6 15:28:49 CEST 2010
Author: bricks
Date: 2010-08-06 15:28:47 +0200 (Fri, 06 Aug 2010)
New Revision: 400
Added:
trunk/bin/listcachedb.py
trunk/bin/publishdebianpackages.py
trunk/bin/treepkginfo.py
trunk/test/test_info.py
trunk/test/test_info_data.py
trunk/treepkg/info/
trunk/treepkg/publish.py
Modified:
trunk/
trunk/bin/publishpackages.py
trunk/demopublishpackages.cfg
trunk/test/test_readconfig.py
trunk/test/test_util.py
trunk/treepkg/packager.py
trunk/treepkg/readconfig.py
trunk/treepkg/util.py
Log:
Merged revisions 346-386 via svnmerge from
svn+ssh://svn.wald.intevation.org/treepkg/branches/treepkg-status
........
r346 | bricks | 2010-07-07 17:40:48 +0200 (Mi, 07 Jul 2010) | 2 lines
added classed for treepkg status xml generation
........
r347 | bricks | 2010-07-08 12:07:39 +0200 (Do, 08 Jul 2010) | 10 lines
renamed new status dir to info because of a naming conflict with status.py
let the user specify a treepkg name in the config
the name is propagated to PackagerGroup
[treepkg]
name: <treepkgname>
becomes:
pg = PackagerGroup(...)
pg.name
........
r348 | bricks | 2010-07-08 15:25:07 +0200 (Do, 08 Jul 2010) | 3 lines
moved test file to test main dir
fixed missing : after def statement
........
r349 | bricks | 2010-07-08 17:59:52 +0200 (Do, 08 Jul 2010) | 2 lines
added treepkg_dir andd tracks_dir attributes to PackageGroup
........
r350 | bricks | 2010-07-08 18:00:32 +0200 (Do, 08 Jul 2010) | 2 lines
changed <millpath> to <trackspath>
........
r351 | bricks | 2010-07-08 18:08:39 +0200 (Do, 08 Jul 2010) | 2 lines
check if config file exists to get usefull error output
........
r352 | bricks | 2010-07-08 18:11:13 +0200 (Do, 08 Jul 2010) | 2 lines
write first info about tracks
........
r353 | bricks | 2010-07-09 12:20:49 +0200 (Fr, 09 Jul 2010) | 2 lines
get all log files
........
r354 | bricks | 2010-07-09 14:42:42 +0200 (Fr, 09 Jul 2010) | 2 lines
added md5sum function
........
r355 | bricks | 2010-07-09 15:15:15 +0200 (Fr, 09 Jul 2010) | 4 lines
implemented nearly all info
only arch and os info are missing
........
r356 | bricks | 2010-07-09 16:19:17 +0200 (Fr, 09 Jul 2010) | 2 lines
Bugfix: source was not defined
........
r357 | bricks | 2010-07-09 16:44:52 +0200 (Fr, 09 Jul 2010) | 3 lines
fixed some typos
added missing add_checksum methond
........
r358 | bricks | 2010-07-09 17:06:15 +0200 (Fr, 09 Jul 2010) | 2 lines
fixed status line for revision
........
r359 | bricks | 2010-07-12 14:10:34 +0200 (Mo, 12 Jul 2010) | 2 lines
let the user specify num revisions that should be shown in the info
........
r360 | bricks | 2010-07-12 14:21:39 +0200 (Mo, 12 Jul 2010) | 2 lines
missed in last commit
........
r361 | bricks | 2010-07-12 17:06:22 +0200 (Mo, 12 Jul 2010) | 2 lines
add os config statement
........
r362 | bricks | 2010-07-12 17:47:42 +0200 (Mo, 12 Jul 2010) | 3 lines
moved packages and logs to revision tag
implemented os and arch info
........
r363 | bricks | 2010-07-13 14:23:28 +0200 (Di, 13 Jul 2010) | 2 lines
updated test readconfig for os config variable
........
r364 | bricks | 2010-07-13 14:24:14 +0200 (Di, 13 Jul 2010) | 2 lines
moved common publish functions to a seperate module
........
r365 | bricks | 2010-07-13 16:26:11 +0200 (Di, 13 Jul 2010) | 2 lines
moved arch info to each package info
........
r366 | bricks | 2010-07-13 18:46:17 +0200 (Di, 13 Jul 2010) | 2 lines
inital checkin for new publishpackages processing
........
r367 | bricks | 2010-07-21 17:52:15 +0200 (Mi, 21 Jul 2010) | 2 lines
implemented parsing from xml string
........
r368 | bricks | 2010-07-22 12:17:42 +0200 (Do, 22 Jul 2010) | 2 lines
copy binary-all packages in all binary-xyz dirs
........
r369 | bricks | 2010-07-22 12:26:40 +0200 (Do, 22 Jul 2010) | 2 lines
fixed small bugs
........
r370 | bricks | 2010-07-22 13:02:36 +0200 (Do, 22 Jul 2010) | 2 lines
added a new root element which is called TreepkgRoot for Treepkg Info classes
........
r371 | bricks | 2010-07-22 13:09:52 +0200 (Do, 22 Jul 2010) | 3 lines
fixed some small issues
remember: run pychecker BEFORE the commit
........
r372 | bricks | 2010-07-22 18:01:43 +0200 (Do, 22 Jul 2010) | 2 lines
improved publishdebianpackages and fixed a lot of bugs in the xml info parsing part
........
r373 | bricks | 2010-07-23 18:40:38 +0200 (Fr, 23 Jul 2010) | 2 lines
implemented CacheDb to store copied Packages from build host
........
r374 | bricks | 2010-07-26 10:28:48 +0200 (Mo, 26 Jul 2010) | 2 lines
moved data handling in publishdebianpackages into seperate module
........
r375 | bricks | 2010-07-26 11:42:07 +0200 (Mo, 26 Jul 2010) | 2 lines
fixed some typos
........
r376 | bricks | 2010-07-26 12:12:40 +0200 (Mo, 26 Jul 2010) | 2 lines
first version of incremental copying debian packages to cachedir
........
r377 | bricks | 2010-07-26 14:58:50 +0200 (Mo, 26 Jul 2010) | 2 lines
incremental copying only changed debian packages
........
r378 | bricks | 2010-07-26 15:39:10 +0200 (Mo, 26 Jul 2010) | 3 lines
use md5 instead of hashlib module
(md5 is deprecated in favour of hashlib in python 2.5)
........
r379 | bricks | 2010-07-26 15:46:19 +0200 (Mo, 26 Jul 2010) | 2 lines
make treepkg compatible to python 2.4
........
r380 | bricks | 2010-07-26 15:50:49 +0200 (Mo, 26 Jul 2010) | 2 lines
fixed last commit (db instead of sqlite3)
........
r381 | bricks | 2010-07-27 10:54:15 +0200 (Di, 27 Jul 2010) | 2 lines
fixed wrong import statement
........
r382 | bricks | 2010-07-27 10:54:34 +0200 (Di, 27 Jul 2010) | 2 lines
removed unnecessary debug output
........
r383 | bricks | 2010-07-27 18:28:22 +0200 (Di, 27 Jul 2010) | 2 lines
check if upload hook is empty
........
r384 | bricks | 2010-07-28 09:35:21 +0200 (Mi, 28 Jul 2010) | 2 lines
run rsync only an arch dirs that are present in the current treepkg
........
r385 | bricks | 2010-07-28 09:35:54 +0200 (Mi, 28 Jul 2010) | 2 lines
add helper tool to list content of a cache db
........
r386 | bricks | 2010-07-28 10:33:48 +0200 (Mi, 28 Jul 2010) | 3 lines
fixed a typo
regex fails if option is empty
........
Property changes on: trunk
___________________________________________________________________
Name: svnmerge-integrated
- /branches/treepkg-status:1-343
+ /branches/treepkg-status:1-343,346-386
Copied: trunk/bin/listcachedb.py (from rev 386, branches/treepkg-status/bin/listcachedb.py)
Copied: trunk/bin/publishdebianpackages.py (from rev 386, branches/treepkg-status/bin/publishdebianpackages.py)
Modified: trunk/bin/publishpackages.py
===================================================================
--- trunk/bin/publishpackages.py 2010-08-06 11:06:08 UTC (rev 399)
+++ trunk/bin/publishpackages.py 2010-08-06 13:28:47 UTC (rev 400)
@@ -20,17 +20,9 @@
from treepkg.run import call, capture_output
from treepkg.cmdexpand import cmdexpand
from treepkg.util import ensure_directory, listdir_abs
+from treepkg.publish import remove_trailing_slashes, expand_filename
+from treepkg.publish import prefix_for_remote_command, copy_to_publishdir
-
-def remove_trailing_slashes(s):
- return s.rstrip("/")
-
-def expand_filename(filename):
- """
- Applies os.path.expanduser and os.path.expandvars to filename
- """
- return os.path.expandvars(os.path.expanduser(filename))
-
config_desc = ["distribution", "section", "num_newest",
"build_user", "build_host", "build_listpackages",
"publish_user", "publish_host",
@@ -74,18 +66,6 @@
" informational output"))
return parser.parse_args()
-def prefix_for_remote_command(user, host):
- """Returns the ssh call needed to run a command on a remote host.
- If host is empty, the function assumes the command is to be run on
- the local host as the same user that exectutes this function, so
- that no ssh or other call is needed.
- """
- prefix = []
- if host:
- prefix.extend(["ssh", "%s@%s" % (user, host)])
- return prefix
-
-
def copy_to_cache(variables, track, revision, arch, quiet):
listpackages_vars = variables.copy()
@@ -126,29 +106,6 @@
call(cmdexpand("scp -p @scp_flags @files $cachedir/", files=files,
scp_flags=scp_flags, **variables))
-
-def copy_to_publishdir(variables, dist, section, arch, quiet):
- destdir = os.path.join(variables["publish_dir"], dist, section, arch)
- remote_destdir = destdir
- if variables["publish_host"]:
- remote_destdir = (("%(publish_user)s@%(publish_host)s:" % variables)
- + remote_destdir)
- runremote = prefix_for_remote_command(variables["publish_user"],
- variables["publish_host"])
-
- call(cmdexpand("@runremote mkdir --parents $destdir",
- runremote=runremote, destdir=destdir, **variables))
- rsync_flags = []
- if variables["publish_remove_old_packages"]:
- rsync_flags.append("--delete")
- if quiet:
- rsync_flags.append("--quiet")
- call(cmdexpand("rsync @rsync_flags -r --perms --times --omit-dir-times"
- " $cachedir/ $remote_destdir/",
- rsync_flags=rsync_flags, remote_destdir=remote_destdir,
- **variables))
-
-
def publish_packages_arch(variables, track, revision, dist, section, arch,
quiet):
copy_to_cache(variables, track, revision, arch, quiet)
Copied: trunk/bin/treepkginfo.py (from rev 386, branches/treepkg-status/bin/treepkginfo.py)
Modified: trunk/demopublishpackages.cfg
===================================================================
--- trunk/demopublishpackages.cfg 2010-08-06 11:06:08 UTC (rev 399)
+++ trunk/demopublishpackages.cfg 2010-08-06 13:28:47 UTC (rev 400)
@@ -59,3 +59,4 @@
# ~user at the beginning of the value and environment variable
# references of the form $VAR or ${VAR} are expanded.
cachedir: /tmp/${LOGNAME}/treepkg-package-cache/%(build_host)s
+cachedb: /tmp/${LOGNAME}/treepkg-package-cache/%(build_host)s-cache.db
Copied: trunk/test/test_info.py (from rev 386, branches/treepkg-status/test/test_info.py)
Copied: trunk/test/test_info_data.py (from rev 386, branches/treepkg-status/test/test_info_data.py)
Modified: trunk/test/test_readconfig.py
===================================================================
--- trunk/test/test_readconfig.py 2010-08-06 11:06:08 UTC (rev 399)
+++ trunk/test/test_readconfig.py 2010-08-06 13:28:47 UTC (rev 400)
@@ -34,6 +34,7 @@
[treepkg]
instructions_file: %(treepkg_dir)s/instructions
check_interval: 3600
+name: testtreepkg
[pkg_simple]
pkg_basename: simple1
@@ -44,6 +45,7 @@
signing_key_id: abcd1234
changelog_msg_template: Update to feature branch r%%(revision)s
version_template: 1.0.svn%%(revision)s
+os: foo os
[pkg_extraargs]
svn_url: svn://example.com/%(name)s/trunk
@@ -55,6 +57,7 @@
orig_tarball: %(base_dir)s/mytarball.tgz
rules_svn_url: file:///tmp/my-debian-repository
deb_build_options: parallel=2
+os: bar os
"""
files = [("treepkg.cfg", config_contents),
@@ -82,7 +85,10 @@
treepkg_opts, packager_opts = read_config(config_file)
self.assertEquals(treepkg_opts,
dict(instructions_file="/home/builder/mill/instructions",
- check_interval=3600))
+ check_interval=3600,
+ name="testtreepkg",
+ treepkg_dir="/home/builder/mill",
+ tracks_dir="/home/builder/mill/tracks"))
self.assertEquals(sorted(packager_opts,
key=operator.itemgetter("name")),
[
@@ -111,7 +117,8 @@
"tracks/extraargs/mytarball.tgz"),
git_url="",
git_branch="",
- builder_cls="PBuilder"),
+ builder_cls="PBuilder",
+ os="bar os"),
dict(name="simple",
base_dir="/home/builder/mill/tracks/simple",
changelog_msg_template=("Update to feature branch"
@@ -133,4 +140,11 @@
version_template="1.0.svn%(revision)s",
git_url="",
git_branch="",
- builder_cls="PBuilder")])
+ builder_cls="PBuilder",
+ os="foo os")])
+
+def main():
+ unittest.main()
+
+if __name__ == "__main__":
+ main()
Modified: trunk/test/test_util.py
===================================================================
--- trunk/test/test_util.py 2010-08-06 11:06:08 UTC (rev 399)
+++ trunk/test/test_util.py 2010-08-06 13:28:47 UTC (rev 400)
@@ -12,7 +12,7 @@
from filesupport import FileTestMixin
-from treepkg.util import replace_in_file, listdir_abs
+from treepkg.util import replace_in_file, listdir_abs, md5sum
class TestReplaceInFile(unittest.TestCase, FileTestMixin):
@@ -72,3 +72,13 @@
self.assertEquals(sorted(listdir_abs(directory, '*.dsc')),
[os.path.join(directory, "foo.dsc")])
+class TestMd5sum(unittest.TestCase, FileTestMixin):
+
+ content = "this is a test content"
+
+ def setUp(self):
+ self.testfile = self.create_temp_file("testmd5.txt", self.content)
+
+ def test_md5sum(self):
+ sum = md5sum(self.testfile)
+ self.assertEquals("a12511153555c1f0f0a1eda200733a3f", sum)
Copied: trunk/treepkg/info (from rev 386, branches/treepkg-status/treepkg/info)
Property changes on: trunk/treepkg/info
___________________________________________________________________
Name: svn:mergeinfo
+
Modified: trunk/treepkg/packager.py
===================================================================
--- trunk/treepkg/packager.py 2010-08-06 11:06:08 UTC (rev 399)
+++ trunk/treepkg/packager.py 2010-08-06 13:28:47 UTC (rev 400)
@@ -323,7 +323,7 @@
return self.build_log + ".gz"
return self.build_log
- def get_log_files(self, logs):
+ def get_log_files(self, logs=None):
files = []
if os.path.isdir(self.log_dir):
for f in os.listdir(self.log_dir):
@@ -331,7 +331,7 @@
f = os.path.join(self.log_dir,f)
if os.path.isfile(f):
files.append((self.get_log_title(f),f))
- return files
+ return files
def list_log_files(self, logs):
"""Returns a list describing the logfiles available for the revision.
@@ -404,7 +404,8 @@
handle_dependencies=False, signing_key_id="", do_build=True,
rules_svn_url=None, deb_build_options="", pkg_basename="",
changelog_msg_template="Update to r%(revision)s",
- svn_subset=(), svn_externals=(), git_branch="", git_url=""):
+ svn_subset=(), svn_externals=(), git_branch="", git_url="",
+ os=""):
self.name = name
# Convert the builder_cls option to a class
@@ -435,6 +436,7 @@
self.do_build = do_build
self.handle_dependencies = handle_dependencies
self.dependencies = None
+ self.os = os
self.pkg_dir_template = "%(revision)s-%(rules_revision)s"
self.pkg_dir_regex = re.compile(r"(?P<revision>[0-9a-f]+)"
r"-(?P<rules_revision>[0-9a-f]+)$")
@@ -642,7 +644,8 @@
def __init__(self, package_tracks, check_interval, revision=None,
instructions_file=None, do_svn_update=True,
- stop_on_error=False):
+ stop_on_error=False, name="", treepkg_dir=None,
+ tracks_dir=None):
self.package_tracks = package_tracks
self.check_interval = check_interval
self.revision = revision
@@ -650,6 +653,9 @@
self.stop_on_error = stop_on_error
self.instructions_file = instructions_file
self.instructions_file_removed = False
+ self.name = name
+ self.treepkg_dir = treepkg_dir
+ self.tracks_dir = tracks_dir
self.sort_tracks()
def sort_tracks(self):
Copied: trunk/treepkg/publish.py (from rev 386, branches/treepkg-status/treepkg/publish.py)
Modified: trunk/treepkg/readconfig.py
===================================================================
--- trunk/treepkg/readconfig.py 2010-08-06 11:06:08 UTC (rev 399)
+++ trunk/treepkg/readconfig.py 2010-08-06 13:28:47 UTC (rev 400)
@@ -9,6 +9,7 @@
import sys
import shlex
+import os.path
from ConfigParser import SafeConfigParser, NoOptionError
import util
@@ -78,12 +79,16 @@
("changelog_msg_template", str, "Update to revision %(revision)s"),
("git_branch", str,""),
("git_url", str,""),
+ ("os", str, ""),
("builder_cls",str,"PBuilder")
]
treepkg_desc = [
("check_interval", int),
"instructions_file",
+ ("name", str, ""),
+ ("treepkg_dir", str, None),
+ ("tracks_dir", str, None)
]
@@ -122,6 +127,9 @@
configuration of the tree packager. The packagers list contains one
dict with the configuratiin for each packager.
"""
+ if not os.path.exists(filename):
+ print >>sys.stderr, "Config file %s does not exist" % filename
+ sys.exit(2)
parser = SafeConfigParser()
parser.read([filename])
@@ -133,8 +141,8 @@
packager_class = parser.get(section, "packager_class", vars=vars)
module = packager.import_packager_module(packager_class)
desc = packager_desc + module.PackageTrack.extra_config_desc
- packager_options = (read_config_section(parser, section, desc,
- defaults=vars))
+ packager_options = read_config_section(parser, section, desc,
+ defaults=vars)
if not packager_options.get("svn_url") \
and not packager_options.get('git_url'):
print >>sys.stderr, "Missing repository URL in section %r" \
@@ -145,8 +153,7 @@
print >>sys.stderr, \
"Warning: git_url in section %r will be ignored" \
% (section)
- packagers.append(read_config_section(parser, section, desc,
- defaults=vars))
+ packagers.append(packager_options)
# main config
treepkg = read_config_section(parser, "treepkg", treepkg_desc)
Modified: trunk/treepkg/util.py
===================================================================
--- trunk/treepkg/util.py 2010-08-06 11:06:08 UTC (rev 399)
+++ trunk/treepkg/util.py 2010-08-06 13:28:47 UTC (rev 400)
@@ -13,6 +13,8 @@
import shutil
import fnmatch
import pwd
+import os.path
+import md5
import run
@@ -156,3 +158,19 @@
def getuser():
"""Returns the login name of the current user owning the proccess"""
return pwd.getpwuid(os.getuid())[0]
+
+def md5sum(filename):
+ """ calculates the md5sum of a file """
+ if not os.path.isfile(filename):
+ raise RuntimeError("Could not create md5sum. File not found: %s"
+ % filename)
+ f = file(filename, 'rb')
+ m = md5.new()
+ while True:
+ d = f.read(8096)
+ if not d:
+ break
+ m.update(d)
+ f.close()
+ return m.hexdigest()
+
More information about the Treepkg-commits
mailing list